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ABSTRACT 



This study examined the role of subcontractors and 
consultants in California's Employment Training Panel (ETP) program. Methods 
included interviews, focus groups, a phone survey, and a survey of projects 
without subcontractors. Results indicated that the market in which 
administrative and training consultants sell their services to employers is 
inefficient, with prices ranging significantly above the competitive and low 
product quality. Over the last 2 fiscal years, the total market for ETP 
subcontractors and consultants was $31-35 million or 36-40 percent of all 
money spent on ETP training. ETP subcontractors and consultants were 
categorized into seven subgroups based on company size and services provided: 
full-service companies; public or nonprofit agencies; hollow companies; niche 
trainers; freelance project administrators; freelance packager; and project 
doctor. Subcontractors and consultants offered benefits and risks to ETP. 

They played a major role in marketing but were aggressive. Many had expertise 
in developing projects, but some driven by profit incentives tried to shape 
projects most profitable to them. They could provide high quality 
state-of-the-art training but had an incentive to reduce the quality and 
hence its costs. They understood policy but tended to manipulate the process 
for gain. Recommendations were made to align the incentives that drive 
subcontractors and consultants with ETP ' s larger objectives and to transform 
the market into an orderly efficient one. (A report on market estimates is 
appended.) (YLB) 
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Executive Summary 



Overview 

In June 1996 the Employment Training Panel (ETP) contracted with a team of researchers in the 
College of Business Administration and Economics at California State University, Northridge to 
conduct a study of the role of consultants in the ETP program. The study was designed to answer 
the following question: 

What should Panel policy be towards subcontractors and consultants 1 who provide 
administrative and training services to employers on ETP proposals and projects? 

Method 

To answer the question posed above, the study team examined the role of subcontractors and 
consultants and analyzed a wide array of data. Project methods included interviews with Panel 
members and senior staff; focus groups with subcontractors, regional staff and employers who 
used subcontractors; interviews with employers and subcontractors and consultants from a 
representative sample of recent projects; and an empirical analysis of recent projects with and 
without subcontractors. 

Market for ETP Consultants and Subcontractors 

Looking broadly at the results of our research, we find that the market in which both 
administrative and training subcontractors and consultants sell their services to employers is best 
described as an inefficient market - one in which prices range significantly above the competitive 
level and the product quality may be low. 

The inefficient market for ETP services has four defining characteristics each of which has 
important implications for ETP. Each characteristic is described below. 

• Low levels of information among buyers: 

This study found surprisingly low levels of information available about subcontractors and 
consultants. This is due in large part to the fact that half the employers do not shop for 
subcontractors and consultants. Our survey found that only half the employers considered more 
than one subcontractor or consultant. 

• Easy entry and exit, multiple paths into the market: 

Over the past three years, there have been approximately 300 subcontractors and consultants 
working on ETP projects, constantly coming in and out of the ETP market. 



1 Throughout the report we will refer to individuals and companies who have an official subcontract on an ETP 
project as subcontractors . We will refer to individuals and organizations who have performed work on an ETP 
proposal or contract without an official subcontract as consultants . 
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Limited likelihood of repeat purchases: 



For most employers, conducting an ETP project is a one time event. It is unlikely that an 
employer will be looking for another ETP project subcontractor or consultant again soon. Hence, 
subcontractors and consultants do not need to be as concerned about repeat business as other 
types of service providers who need customers to make repeat purchases to be successful. 

• Third party funding: 

A problem unique to the hiring of ETP subcontractors and consultants is the fact that employers 
may not treat ETP money like their own money. 

Results 

We estimate that the total size of the market for subcontractor and consultant services generated 
by ETP projects in fiscal years 1995 and 1996 was between $31,191,000 and $34,855,00 or 
between 36% and 40% of all funds spent on ETP training. 

ETP subcontractors and consultants can be broken into seven subgroups based on the size of the 
company and the services they provide: 

• Full Service Companies (20%) : These companies provide a full range of ETP services 
including proposal development, administration, and a range of training programs. 

• Public or Non-profit Agencies (5%): This category includes a wide range of public and 
private non-profit agencies. 

• Hollow Companies (25%) : These companies market a full range of ETP services but are in 
fact small companies that subcontract out for most of the services they provide to a network 
of other small firms or individuals. 

• Niche Trainers (20%): These companies are small highly specialized training companies, 
usually with few employees. They provide specialized training in narrow market niches. 

• Free-lance Project Administrators (20%) : This group is made up of individuals and 
occasionally partnerships, who primarily provide administrative services, often one project at 
time. 

• Free-lance Packager (10%): This group is made up of individuals and occasionally 
partnerships whose primary business is seeking out companies who may be interested in ETP 
training, and helping them develop a proposal that will be accepted by the Panel. 
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• Project Doctor (<1%): Much like Hollywood script doctors, these subcontractors and 
consultants often become involved in a project after it has run into trouble. They tend to 
specialize in solving administrative problems and negotiating with ETP. 

Overall, most employers surveyed believed the subcontractors and consultants they hired played 
a key role in the success of their project. Most employers remain satisfied with the 
subcontractors and consultants they choose. There does seem to be a problem group of 
subcontractors and consultants however. We estimate from this survey that between 10% and 
20% of the subcontractors and consultants on the projects we studied did not perform up to the 
expectations of the employer. 

Interestingly, when we cross-tabulated the evaluation of the subcontractor with whether or not 
the employer considered more than one subcontractor, we found that all the employers who were 
dissatisfied had not shopped for a subcontractor, but considered only one. This indicates to us 
that a lack of information about alternatives does lead to less satisfaction with the choice of 
subcontractor. 

Clearly the most important benefit employers got from using a subcontractor was help in 
managing what the employers saw as the complex ETP process. Fifty-one percent of the 
employers gave the benefit related to managing the program as the most significant benefit 
received from using a subcontractor, while only 30% identified the benefit related to training as 
the most significant. Fully one-quarter of the employers reported that help keeping track of 
complex ETP “rule and policy changes” was the most significant benefit. An additional 17?/o 
said “handling ETP paper work the employer’s staff could not,” and 1 1% believed a 
subcontractor” made it easier to get ETP to approve the project.” Conversely, 19% said “provide 
trainers employer did not have in-house” and 1 1% reported “designed custom training” as the 
most significant benefit. 

Impact of Subcontractors and Consultants on ETP 

Since subcontractors’ and consultants’ most severe critics will admit some subcontractors and 
consultants provide some valuable services and since the most ardent defenders of subcontractors 
and consultants admit there are problem subcontractors and consultants out there, we have 
organized our summary around various aspects of the ETP program and attempted to 
demonstrate how subcontractors and consultants offer a benefit to ETP, and contrast this with 
the risks that they pose to the program. 

Marketing 

Benefits : This is the area with the strongest consensus. All the groups we interviewed 
recognized that subcontractors and consultants play a major role in marketing ETP. One senior 
ETP manager estimates that they are responsible for bringing in as much as 80% of all ETP 
projects. 
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Risks : Subcontractors and consultants are aggressive, they will test the boundaries or push the 
envelope on ETP policies to try and qualify projects which are marginal, look for loopholes to 
slip through projects that may conform to the letter of ETP policy but not to the spirit of ETP 
priorities. 

Developing Projects 

Benefits : Many of the training and management companies that do ETP subcontracting and 
consulting have a wealth of specialized expertise in skill areas, in industries, and managing 
projects. Good subcontractors and consultants help companies do effective needs assessment 
that links training to larger corporate goals and they deliver high quality state-of-the-art training 
that is customized for the individual companies. 

Risks : Some subcontractors and consultants driven by powerful profit incentives will try to 
shape projects to be most profitable for them. 

Managing Projects 

Benefits : Effective subcontractors and consultants can save monitoring staff time by knowing 
ETP procedures and providing needed documentation and information promptly. 

Subcontractors help implement ETP policy changes by tracking changes and keeping employers 
up-to-date. 

Risks : Subcontractors and consultants may distort ETP policies and undermine effectiveness, by 
testing boundaries, seeking loopholes in order to serve existing clients or generate more profits. 

Delivering Training 

Benefits : Today, even relatively large companies may have few or no in-house trainers. Training 
subcontractors and consultants can provide ETP programs with high quality state-of-the art 
curricula and trainers. Without training subcontractors and consultants, companies without an in- 
house training capacity simply could not participate in ETP. 

Risks : Training subcontractors and consultants have an incentive to reduce the quality of 
training and hence its costs in order to reap larger profits. 

Policy Process 

Benefits : Subcontractors arid consultants spend a lot of time working with employers and the 
ETP process, hence they have extensive and detailed knowledge of how ETP policies are 
operationalized in the field. Tapping this understanding, as this project tried to do, could provide 
the Panel with valuable information for policy development. 

Risks : Once again, as profit driven entities, subcontractors and consultants are likely to try to 
manipulate the policy process solely for their own gain. 
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Recommendations 



Our recommendations have two overarching goals: (1) to align the incentives that drive 
subcontractors and consultants with the larger objectives of the Panel and (2) to transform the 
inefficient market for ETP subcontractors and consultants into an orderly efficient market. To 
accomplish these goals, we recommend that the Panel act to increase the amount of information 
available about subcontractors and consultants and take other steps to help employers make 
informed choices about subcontractors and consultants. 

1. The Panel should provide employers with an easily accessible system of consumer 
information on subcontractors. 

We recommend that the system include the following types of information: 

• Name and contact information for all subcontractors who have been listed as 
subcontractors on ETP projects in the last three years. 

• The names and contact information of ETP clients each subcontractor has had over 
the last three years. 

• Descriptive information of the subcontractor’s work on the projects, including: 
project size, industry, type of training or administrative services delivered. 

• Information on the fees paid to subcontractors for services provided including: 
proposal development, administration, and training. 

• Percent of planned trainees who completed training on each project. 

• A rating of the subcontractor’s performance by the employer. 

• Subcontractor’s attendance at recent ETP training sessions. 

2. The Panel should develop a consumer guide to help employers choose 
subcontractors and consultants wisely. 

3. The Panel should provide semi-annual training for subcontractors and post 
attendance on the information system. 

4. The Panel should work with subcontractors to get advice and insights about issues 
related to ETP. 

5. The Panel should consider how subcontractors and consultants shape the 
consequences of ETP policies, and conversely how ETP policies drive 
subcontractor or consultant behavior when developing policies. 
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Introduction 

In June 1996 the Employment Training Panel (ETP) contracted with a team of researchers 
in the College of Business Administration and Economics at California State University, 
Northridge to conduct a study of the role of subcontractors and consultants in the ETP 
program. The study was designed to answer the following question: 

What should Panel policy be towards subcontractors and consultants who provide 
administrative and training services to employers on ETP proposals and projects? 

Managing the role of subcontractors and consultants' in ETP projects has been a 
continuing challenge for the Panel. The Panel funded an earlier project on the role of 
subcontractors (previously including and referred to as consultants) in 1989 indicating that 
the role of subcontractors and consultants has been controversial for at least that long. 

Panel staff are concerned that subcontractors and consultants may undermine the 
effectiveness of ETP in a variety of ways which we discuss later. From our perspective, it 
seems that ETP staff, close observers of ETP, and even the subcontractors and consultants 
themselves would agree that there are high quality, effective subcontractors and consultants 
who enhance the delivery of ETP training, and at the same time, there are problem 
subcontractors and consultants who undermine the effectiveness of Panel policies and 
training. This view is supported by the results of the earlier study. For us, the issue for 
the Panel is how to manage subcontractors and consultants in the ETP program, so that 
they contribute to the larger objectives of ETP. 

To answer the question posed above, the study team examined the role of subcontractors 
and consultants and analyzed a wide array of data. Project methods included interviews 
with Panel members and senior staff; focus groups with subcontractors, regional staff and 
employers who used subcontractors; interviews with employers and subcontractors and 
consultants from a representative sample of recent projects; and an empirical analysis of 
recent projects with and without subcontractors. (The study’s specific methods are 
reviewed in detail in a later section.) 

We begin by reviewing the results of the earlier study, and presenting our project’s 
objectives in more detail. We then review the methods we employed. Next, we present 
our central finding which is that both employers and subcontractors and consultants 
operate in a “Inefficient Market” which has created many of the problems identified with 
subcontractors and consultants. We estimate the size of the markets for subcontractors and 
consultants and describe who the subcontractors are and where they come from. We 
describe the mix of services provided by subcontractors and consultants. Then we turn our 
attention to the Panel and Panel staffs perception of the subcontractors and consultants, 
and the subcontractors’ and consultants’ view of the staff and Panel. We wrap up the 
results section with a complete review of the benefits and risks posed by subcontractors 



1 See page 2 for definition of subcontractor and consultant. 

2 Wilms, Wellford and Moore, Richard W. Brokering G overnment Services: An Analysis of Administrativ e 
Consultants and the California Employment Training Panel . Sacramento: Employment Training Panel, 
September 12, 1989. 
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and consultants’ participation in the program. We conclude with a series of 
recommendations based on the concept of creating a more orderly and informed market for 
the selection of ETP subcontractors and consultants. 

Perceived Problems with Subcontractors and Consultants 

Many Panel staff members and some close observers of ETP have come to associate the 
participation of subcontractors and consultants in ETP with a variety of problems. 
Specifically some argue that the problem ETP has in having to disencumber a large 
percentage of the funds 3 it has contracted to expend is caused by subcontractors and 
consultants “over selling” employers, encouraging them to apply for more training then 
they can practically accomplish. A second criticism of subcontractors and consultants is 
that they encourage the participation of employers who have substantial regular training 
programs and whose workers are not really at risk of layoff to apply for ETP training, by 
selling ETP as “free government money.” Subcontractors and consultants use their 
detailed knowledge of ETP procedures to get these proposals approved and ETP ends up 
subsidizing training that would have occurred anyway. There are accusations that some 
subcontractors and consultants have found ways to reap excessive profits. Excessive 
profits are generated by getting a naive employer to purchase a training program that is 
essentially an off-the-shelf curriculum, taught by low paid instructors, and then getting paid 
at ETP’s fixed fees which assume more customized training and more expensive 
instructors, thereby reaping a large profit and producing little of value. Overall some 
individuals see these abuses by subcontractors and consultants as a threat to ETP’s long- 
term survival. 

Evaluating these concerns has been difficult for Panel members and staff. There are little 
systematic data on subcontractors and consultants and their practices. Many consultants 
work on ETP projects but do not sign and submit formal subcontracting agreements with 
ETP. Thus there is a substantial population of “hidden consultants” also known as 
“phantom consultants,” and ETP knows even less about this group’s behavior. 

Before we proceed a note on terminology. Throughout the report we will refer to 
individuals and companies who have an official subcontract on an ETP project as 
subcontractors . We will refer to individuals and organizations who have performed work 
on an ETP proposal or contract without an official subcontract as consultants . We 
recognize that the same individuals or organizations may have been a “subcontractor” at 
one time and a “consultant” at another time. 

Related Research 

Any time government sets up a system to award contracts or grants, brokers who have 
special knowledge of the process, spring up to help applicants win the contracts and grants. 
We searched the published academic literature in the fields of political science, economics, 



3 Historically about 40% of the money ETP has awarded in contracts has not been spent on the contract, and 
thus disencumbered. 
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and public administration for previous policy studies on government policy towards these 
types of brokers. Surprisingly, we found none. 

We begin our analysis of ETP subcontractors and consultants by reviewing the findings of 
the earlier study of subcontractors (previously all called consultants) cited before. It is 
important to note that this study looked exclusively at administrative subcontractors and 
did not consider training subcontractors. 

The study had four primary conclusions 4 : 

• there were no measurable differences between the characteristics of projects 
with and without subcontractors, 

• Panel operations and regulations create the demand for subcontractors, 

• most employers are satisfied with the subcontractors they chose, 

• the positive impact of ETP training is the same on projects with and without 
subcontractors. 

Based on their findings Wilms and Moore made five recommendations to the Panel 5 : 

1 . There is no evidence of a need to regulate subcontractors. 

2. The Panel should closely monitor the costs and benefits of supporting 
administrative subcontractor costs. 

3. Panel members should be aware that their own regulations govern the market 
for subcontractors. 

4. Panel members and staff need to be scrupulous in their dealings with 
subcontractors. 

5. The Panel should make clear to the business community that engaging 
subcontractors in no way implies an advantage in securing Panel funding. 

Much has changed since this study was completed. ETP has increased its focus on 
economic development and gone to a policy of targeting basic industries only, by 
emphasizing training for businesses faced with out-of-state competition and training to 
support the State’s export base. The size and complexity of the program has grown 
substantially. In turn, the number of subcontractors and consultants working in ETP 
projects has increased significantly. 

Research Questions 

Based on the Panel’s history with subcontractors and consultants and the larger policy 
question posed above, we designed a study to answer the following research questions: 

1 . What are the characteristics of subcontractors and consultants who are involved 
in ETP contracts? 

2. What roles do subcontractors and consultants play in the ETP process? 

3. Why do some consultants prefer to remain hidden? 



4 Wilms and Moore, pg. 3 1. 

5 Wilms and Moore, pages 34-37. 
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4. How large is the market for ETP subcontractor and consultant services both 
open and hidden? 

5. Why and how do employers choose subcontractors and consultants, and how do 
they evaluate the subcontractor’s or consultant’s performance? 

6. What impact have subcontractors and consultants had on the ETP process? 

7. What impact do subcontractors and consultants have on ETP policy? 

8. How have other similar public agencies handled the issue of subcontractors and 
consultants? 

Methods 

To answer the questions posed above, we conducted individual in-person, and individual 
phone interviews with Panel members, and ETP executive staff. We conducted focus 
groups with ETP field staff, ETP subcontractors, and employers who had an ETP project 
in the last fiscal year (July 1995-1996). In addition, we conducted a phone survey of a 
sample of half the projects which reported using subcontractors that ended in FY 1996. In 
the survey, we interviewed both the employer and the primary subcontractor. We also 
surveyed all of the projects from the last fiscal year which did not report an official 
subcontractor to determine how many had used a consultant. We assembled a database of 
existing ETP data on projects that closed in FY 1994-95 and 1995-96 to estimate the size 
of the subcontractor and consultant market and to describe the differences between projects 
with and without subcontractors. 

Interviews 

* Panel Member Interviews 

The purpose of Panel member interviews was to obtain their views on the role 
subcontractors and consultants play in the ETP program and on policy issues related to 
subcontractors and consultants. 

Panel members answered questions about past and present roles of subcontractors and 
consultants in the ETP program; the impact subcontractors and consultants have had on the 
development of proposals, delivery of training, and shaping of ETP policy; what problems 
the Panel has encountered with subcontractors and consultants; Panel members' views on 
subcontractors and consultants and what the Panel believes the policy for subcontractors 
and consultants should be. 

We interviewed six of eight Panel members, individually by phone, between October 1996 
and November 1996. 

* ETP Executive Staff 

The purpose of executive staff interviews was to obtain their views on the role 
subcontractors and consultants play in the ETP program, and on policy issues related to 
subcontractors and consultants. 
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Executive staff answered questions about past and present roles of subcontractors and 
consultants in the ETP program; the impact subcontractors and consultants have had on the 
development of proposals, delivery of training, and shaping of ETP policy; what problems 
the executive staff has encountered with subcontractors and consultants; Executive staffs 
views on subcontractors and consultants and what they believe Panel policy should be for 
subcontractors and consultants. 

We interviewed all ETP executive staff members, individually in person, at ETP 
Sacramento offices, in August 1996. 

Focus Groups 

* ETP Field Staff 

The purpose of the ETP staff focus groups was to collect information about the staffs 
experience with subcontractors and consultants and their assessment of the impact of 
subcontractors and consultants on ETP programs. 

ETP field staff discussed how the role of subcontractors and consultants has evolved over 
time, how subcontractors and consultants help ETP achieve its objectives, and what each 
of the various types of subcontractors and consultants has contributed to the success (or 
failure) of projects. Staff also discussed at length problems they had encountered working 
with subcontractors and consultants. They also made suggestions about how ETP can 
better manage subcontractors and consultants. 

We conducted two focus groups with ETP field staff in the San Mateo office (six 
members) and the North Hollywood office (eight members). 

* Subcontractors 

The purpose of the subcontractor focus groups was to collect subcontractors' views on their 
own work in general and their role in the ETP program, and to solicit ideas for 
improvements in the ETP program. 

In the focus groups, subcontractors discussed their backgrounds, the various ways they 
obtain clients (for both ETP projects and others) and how they became involved in ETP 
projects. They spoke of the problems some subcontractors and consultants cause for ETP 
projects. Other topics included: fixed fees, agency agreements, risk sharing with 
employers, various ETP regulations and monitoring practices, the probable impact of the 
ETP policy that requires employers to reveal fees paid to consultants who help develop the 
proposal, and alternative policies for managing subcontractors and consultants in the ETP 
process. 

We conducted two focus groups with ETP subcontractors, one with six subcontractors (of 
seven who were to be there) at the University Club on the California State University, 
Northridge campus, and one with seven subcontractors (of seven who were to be there) at 
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the San Mateo ETP office. Participants were chosen to be representative of the 
subcontractors in that region. 

• Employers with Subcontractors 

The purpose of the employer focus groups was to collect employers' views on the role their 
subcontractors played in specific ETP projects, and to solicit ideas for improvements in the 
ETP program. 

In the focus groups, employers answered questions about how they first became involved 
with the ETP, why they used an ETP subcontractor, and how well the ETP project 
achieved their objectives. Other topics included: ETP regulatory demands, monitoring 
processes, ETP paperwork, and ways to manage subcontractors in the ETP program. 

We conducted two focus groups with employers, one with two employers (of seven who 
were to be there) at the University Club on the California Sate University, Northridge 
campus, and one with three employers (of six who were to be there) at the San Mateo ETP 
office. Participants were chosen to be representative of employers in that region. 

Phone Surveys 

• Sample 

We randomly selected half (36) of the projects that closed in FY 1996 that reported using a 
subcontractor on the project for a phone survey. We began by contacting the employer on 
the project. If the company was unreachable, had gone out of business or was a consortium 
project, we replaced the project with a second project. 

In the employer interview, we asked the employer to identify the primary subcontractor on 
the project. We then interviewed the subcontractors. We completed interviews with 33 
employers-a response rate of 94%, and 32 subcontractors-a response rate of 91%. 

• Interview Guides 

Interview guides including both structured and open-ended questions were developed by 
project staff based on the results of the focus groups. Drafts of the instruments were 
reviewed by ETP staff and refined. 

The employer interview guide included questions about how employers first learned about 
ETP and why their ETP project was created, how they chose their ETP subcontractor and 
for which aspects and tasks of the project they used their ETP subcontractor. Other topics 
included: compensation for the subcontractor and opinions about a third-party rating 
system to provide employers information about the quality of subcontractors and 
consultants. 

We used a similar approach to create an interview guide for the subcontractors. The 
interview guide included questions about their background and their company, their 
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relationship with the ETP employer, their marketing methods, their involvement in the 
ETP project, why the employer used them, and how they were compensated for the project. 
They also answered open-ended questions about why some consultants prefer to be paid 
outside the ETP contract and what they think about a third party-rating system for 
employers on subcontractors. 

* Analysis 

Data from the questionnaires were reviewed. Open-ended questions were categorized and 
coded. A statistical analysis of the data was completed. 

Survey of Projects without Subcontractors 

* Sample 

We interviewed all employers from projects in fiscal year 1996 who did not report an 
official subcontractor. Of the 38 projects without a reported subcontractor, we were able to 
contact 37. 

* Interviews 

The purpose of the interviews was straightforward and we asked four simple questions: (1) 
had the employer used a consultant to prepare the ETP proposal? (2) had the employer 
used a consultant to administer the ETP project? (3) had the employer used a consultant to 
provide training for the project? (4) had the employer hired any employees to work on the 
ETP project? 

* Analysis 

Data from the survey were used to estimate the size of the consultant market. 

Results 

The Inefficient Market for ETP Subcontractors and Consultants 

Looking broadly at the results of our research, we find that the market in which both 
administrative and training subcontractors and consultants sell their services to employers 
is best described as an “Inefficient Market. 6 ” This is a market with prices often above the 
competitive level and with possible large variations in product quality. 

One hallmark of efficient markets is widely-shared consumer information on the various 
sellers’ prices and product qualities. Widely-shared information allows the consumers to 
comparison shop and to purchase where they obtain the best combination of price and 
quality. New and developing markets initially lack the information to identify the best deal 
and consequently prices and qualities may vary widely. But, as product price and quality 



6 See for example Jeffrey M. Perloff and Klaas T. van’t Veld, Modem Industrial Organization, second 
edition, Harper Collins College Publisher, 1994, pp. 554 ff. 
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information becomes more widely available, the relatively high-priced, low-quality 
producers are unable to make sales, and disappear from these markets leaving the 
producers of the desired price/quality goods. Newcomers in this market will have to offer 
products with prices and qualities that meet or beat the existing producers, otherwise the 
new producers will not be able to sell in this market. The essential ingredient in an 
efficient market is the widespread information about the Sellers, their locations, their 
prices, and product qualities. 

Economists have investigated the effects of the absence of quality and price information on 
the functioning of markets and have found that information-limited markets operate 
inefficiently compared to information-rich markets. The absence of product quality 
information leads to lower-quality products dominating the market because the consumers 
who cannot distinguish variations in the quality of the product are not willing to pay a 
higher price for a higher quality product. The absence of a higher remuneration to the 
companies producing the higher quality products eventually drives those products from the 
market. 

Limited price information can result in several market outcomes depending on the nature 
of the price information available. If consumers initially lack any price comparison 
information but can search for it among known sellers at a cost, the result is either an 
equilibrium price at the monopoly-level price or the absence of an equilibrium price 
altogether. The absence of an equilibrium price results from the incentive of sellers to 
raise prices above the competitive (efficient) level because consumers incur costs to search 
for a lower price. This means that the sellers who raise prices will not lose all of their sales 
and their profits will rise, continuing their incentive to raise prices further. This may lead 
other sellers, particularly new entrants, to cut prices from the monopoly level to gain entry 
and business. In this case, there is a dispersion of prices and individual sellers are 
constantly adjusting their prices. When a market has a mixture of informed and 
uninformed consumers it will produce a multi-price equilibrium with the informed 
consumers paying the competitive price and the uninformed shopping randomly, 
encouraging some sellers to price at the highest level that the uninformed consumer is 
willing to pay. 

In a situation that closely parallels the ETP-generated market for consultants and 
subcontractors, if consumers are aware of some of the sellers but not others, the market is 
likely to have a full range of prices charged. Unfortunately, the prices charged will be 
unrelated to the qualities of the services; rather the price paid by any one customer will be 
more closely related to that customer’s knowledge of other sellers’ prices and the buyer’s 
incentive to search out a good deal, along with the costs of doing so. Curiously, in this 
case, the entry of new sellers into the market can actually raise the average price charged 
by increasing the search cost for the customers . 7 Lowering the cost of searching for a 
lower-cost provider in these markets is likely to lead to a lower average price and less price 
dispersion. 



7 Usually seller’s easy entry and exit from a market guarantee an efficient market and a competitive price 
because sellers who charge more than a competitive price will be punished by competition from new entrants 
and will lose virtually all of their sales and profits. 
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The inefficient market for ETP services has four defining characteristics each of which has 
important implications for ETP. Each characteristic is described below with its 
implications for ETP policy. 

* Low levels of information among buyers: 

This study, like the earlier study, found surprisingly low levels of information available 
about consultants and subcontractors. This is due in large part to the fact that half the 
employers do not shop for consultants and subcontractors. Our survey found that only half 
the employers considered more than one consultant or subcontractor. A third of the 
employers got their consultant or subcontractor through a cold call. Most employers 
evaluated consultants and subcontractors exclusively on information provided by the 
consultant or subcontractor. There is, in fact, no readily available data on consultant’s or 
subcontractor’s experience or price. Employers cannot open an industry publication or the 
yellow pages and find information on consultants’ and subcontractors’ prices and services 
listed side by side to help them choose wisely. 

We also found that, as a business strategy, consultants and subcontractors try to develop 
niche markets and dominate them exclusively. For example, a consultant or subcontractor 
may specialize in TQM training for union manufacturers in one industry. If he or she can 
dominate the market, it discourages other consultants and subcontractors from attempting 
to enter the niche, which leaves employers in this sector with few alternatives and little 
information. This situation means that many employers will continue to make decisions to 
hire consultants and subcontractors with little objective information about their 
qualifications or their prices. 

* Easy entry and exit, multiple paths into the market: 

Over the past three years, there have been approximately 300 consultants and 
subcontractors working on ETP projects, constantly coming in and out of the ETP market. 
This is because it is very easy to enter the market for ETP consulting, and even easier to 
leave the market. Many consultants and subcontractors work on a single project and then 
never work on another. There is no license required, and no clear standards to direct 
employers to one consultant or subcontractor over another. The results of our survey, that 
follow this section, clearly show that ETP consultants and subcontractors come from a 
wide range of backgrounds. Some consultants and subcontractors enter the ETP market 
with extensive experience with ETP, some being former ETP staff. Some enter the field 
with long experience in management consulting or training. Others have little or no first- 
hand experience with ETP. 

The consequence of these phenomena for ETP is that staff must contend with a constantly 
shifting array of consultants and subcontractors. While some consultants and 
subcontractors work on many projects over many years, most consultants and 
subcontractors are involved briefly and leave, or do ETP projects very sporadically. This 
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makes the development of on-going relationships and communication between ETP and 
the consultants and subcontractors difficult at best. 

• Limited likelihood of repeat purchases: 

For most employers, conducting an ETP project is a one time event. It is unlikely that an 
employer will be looking for another ETP project consultant or subcontractor again soon. 
Hence, consultants and subcontractors do not need to be as concerned about repeat 
business as other types of service providers who need customers to make repeat purchases 
to be successful. This in turn reduces the incentive to keep the customer satisfied. This 
phenomenon is less of a problem for consultants and subcontractors who provide non-ETP 
services to clients, since they do have a market for repeat business outside of ETP. 

Also, since employers are likely to only choose an ETP consultant or subcontractor once, 
they lack experience, making them fairly unsophisticated consumers. Unless an employer 
has a contact at a similar company that has conducted an ETP project, or has a staff person 
who is familiar with ETP, they will have little opportunity to gain the information they 
need to evaluate consultant’s or subcontractor’s claims. Our survey data show that most 
employers are selecting a consultant or subcontractor without any objective information 
about the consultant’s or subcontractor’s performance. 

• Third party funding: 

A problem unique to the hiring of ETP consultants and subcontractors is the fact that 
employers may not treat ETP money like their own money. An employer who has not fully 
thought through the costs of training may perceive that ETP is paying all of the direct costs 
of training, and hence they may not be as vigilant in selecting a consultant or subcontractor 
or in driving down the price as the employer would be if spending money from the 
company’s own budget. The analogy here would be an insured consumer whose car has 
been damaged in an accident and is allowed to select any repair shop he/she wants. The 
consumer wants a good job done, but is less concerned about the price than he/she might 
be otherwise. 

Toward More Efficient Markets 

Governments often encourage the spread of information in inefficient markets. Examples 
include the requirement that gas stations post their pump prices so that motorists may 
comparison shop as they drive by, and financial institutions’ requirement to advertise 
interest rates on a common basis (annual percentage yield) to facilitate comparison 
shopping. In the recommendation section we suggest a variety of strategies which we 
believe will help create an efficient market for ETP subcontractors and consultants. 

Market characteristics that promote the availability of information to consumers include: 

• obvious benefits to the customer from obtaining comparison price information, 

• low-cost, centralized information sources, 
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• easily-identified sellers of the product or service, and 

• frequent repeat purchases by consumers. 

Size of the Subcontractor and Consultant Market 

The first task in estimating the size of ETP’s contribution to the subcontractor and 
consultant market was to estimate the impact of ETP’s expenditures on the employee 
training market in California. Initially, it would seem that the expansion of the training 
market due to the existence of ETP would be best estimated by the value of its training 
contracts. The 394 ETP contracts that were completed during fiscal 1994-95 and 1995-96 
originally totaled $145,140,000 but were amended down to $136,725,000 during the 
training process. However, the contractors only earned $74,985,000 under these 394 
contracts because a number of the projected trainees did not complete training (which 
requires placement and employment retention for at least 90 days in a training-related 
position). ETP’s impact on the training market in California probably lies somewhere 
between that total contracted amount and the actual earnings of the ETP contractors. Our 
best estimate, which is derived and fully discussed in Appendix A, is that the existence of 
ETP added an additional $86,3 10,000 to employee training efforts in California. 



ETP Training Market FY 1994-95 and 1995-96 
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Estimating the size of the subcontractor and consultant market is a complex task. We 
made two estimates of the size of the subcontractor and consultant market for fiscal years 
1995 and 1996. A maximum estimate indicating the largest likely size of the subcontractor 
and consultant market, and a minimum estimate. In estimating the market, we considered 
the amount of training for which ETP contracted, reduced it to the amount ETP actually 
paid for, and estimated additional employer expenditures that would not have occurred 
without ETP funding. We estimated the size of the consultant market by taking the 
amount of funding for training and administrative subcontractors in ETP records, estimated 
the frequency of consultants without an official subcontract, and adjusted these dollar 
amounts for disencumbrance rates. A complete report on our calculations is included in 
Appendix A. Table 1 reports our key findings. 
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Table 1 

Estimated Market for ETP Training and Estimated Market Shares for 
Subcontractors and Consultants 
(Fiscal Years 1994-95, 1995-96) 





Maximum 

Estimate 


Minimum Estimate 


Market Segment 


Dollar 

Value 


% of 
Market 


Dollar 

Value 


%of 

Market 


Estimated Total ETP Training Market 


86,310,000 


100.0 


86,310,000 


100.0 


Estimated Administrative Subcontractors 


2,635,000 


3.0 


3,024,000 


3.5 


Estimated Training Subcontractors 


28,110,000 


32.6 


[ 24,447,000 


28.3 


Estimated Administrative Consultants 


513,000 


.6 


589,000 


.7 


Estimated Training Consultants 


3,617,000 


4.2 


3,146,000 


3.6 


Total Estimated Subcontractor and Consultant 
Market 


34,875,000 


40.4 


31,206,000 


36.1 



As the table indicates, we estimate that over the last two fiscal years the total market for 
ETP subcontractors and consultants was between about $35 million and $3 1 million, or 
between 40% and 36% of all money spent on ETP training. Subcontractors make up the 
bulk of the market at 88%, while the much discussed consultants accounted for 12% of the 
market. 



Training subcontractors and consultants are the largest part of this market by far, 
accounting for around 34% of the total ETP market. Administrative subcontractors’ and 
consultants’ market share is slightly below 4% by our estimates, which seems surprisingly 
small given that ETP budgets 1 5% for administrative costs. 



It is important to note that these estimates include all 394 contracts let by ETP during fiscal 
years 1994-95 and 1995-96. The remainder of the analysis in this report excludes contracts 
with consortia and training agency trainers (like community colleges and other training 
providers), as well as contracts that were subsequently terminated. 
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Differences Between Projects With and Without Subcontractors 



To analyze the differences between projects that did and did not use subcontractors we 
took the 260 projects from fiscal years 1994-95 and 1995-96 which were not consortia or 
training agency projects or subsequently terminated and divided them into four groups: 

• contracts with both training and administrative subcontractors, 

• contracts with only training subcontractors, 

• contracts with only administrative subcontractors, 

• contracts with no subcontractors. 

As the graph indicates during this period 79% of the contracts reported a subcontractor. 



Fewer than one-third of the contracts used both a training and administrative subcontractor, 
while 44% used a training subcontractor only and four percent used only an administrative 
subcontractor. 

Using data from ETP contract files we compared these groups on a variety of 
characteristics. Table 2 summarizes this comparison. Most of the data definitions in Table 
2 are obvious from the data labels but the following few are not. Under “Size of Business” 
a small business is defined as one with 1 to 100 employees, a medium-sized business has 
101 to 250 employees, and a large business has more than 250 employees. The designated 
“Office” is the office in which the contract was developed; all contracts are monitored and 
audited out of the Sacramento and Los Angeles offices. Table 2 identifies three types of 
“Skills Training”. Basic Skills includes vocational English as-a-second-language (VESL) 
and basic mathematics. Soft Skills include total quality management (TQM); just in time 
(JIT) delivery; teamwork, leadership, and communication (TLC) training; sales techniques, 
and customer Service. Hard Skills include a long list of specific job skills such as asbestos 
removal, auto repair, airplane mechanic, electrician, carpentry, licensed vocational nurse, 
meat cutting, and welding among others. 



Type of Subcontractor Used 
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Only 
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Table 2 

Characteristics of FY 1995 and FY 1996 
Non-consortia/Training Agency Contracts By Subcontracting Group 





All Contracts 


Contracts with Contracts with 


Contracts with 


Contracts 








Both Training & Only Training 




Only 


without 








Administrative Subcontractors 


Administrative 


Subcontractors 








Subcontractors 




Subcontractors 




Count 




260 




80 


114 




11 


55 


Final Contract Amount 


$ 


285,265 


$ 


293,627 $ 


210,306 


$ 


591,127 


$368,819 


Original Contract Amount 


$ 


312,295 


$ 


314,585 $ 


236,586 


$ 


597,922 


$410,550 


Final/Original Contract Amount 




91.3% 




93.3% 


88.9% 




98.9% 


89.8% 


Amount Earned 


$ 


168,742 


$ 


178,324 $ 


126,562 


$ 


272,070 


$222,546 


Amount Disencumbered 


$ 


116,523 


$ 


115,303 $ 


83,744 


$ 


319,056 


$146,273 


Percentage Disencumbered 




40.8% 




39.3% 


39.8% 




54.0% 


39.7% 


Size of Business 


















Small 




25.9% 




18.7% 


36.8% 




0.0% 


20.0% 


Medium 




18.5% 




22.5% 


21.1% 




0.0% 


10.9% 


Large 




55.6% 




58.8% 


42.1% 




100.0% 


69.1% 


Type of Business 


















Manufacturing 




63.3% 




73.8% 


64.0% 




18.2% 


54.5% 


Services 




30.5% 




18.8% 


29.8% 




63.6% 


41.8% 


Other 




6.2% 




7.4% 


6.2% 




18:2% 


3.7% 


Original Contracted Placement 




188 




189 


149 




360 


236 


Final Contracted Placement 




186 




184 


143 




358 


242 


Number Placed 




110 




108 


87 




165 


150 


Training Subcontract Amount 






$ 


140,386 $ 


111,395 








Training Subcontract Percentage 








47.8% 


53.0% 








Administrative Subcontract 






$ 


27,979 




$ 


61,453 




Amount 


















Administrative Subcontract 








9.5% 






10.4% 




Percentage 


















Los Angeles Office 




50.8% 




57.5% 


48.2% 




45.5% 


47.3% 


Sacramento Office 




25.4% 




16.3% 


26.3% 




27.3% 


36.4% 


San Mateo Office 




23.1% 




26.3% 


24.6% 




27.3% 


14.6% 


San Diego Office 




0.1% 




0.0% 


0.9% 




0.0% 


1.8% 


Soft Skills Training 




89.2% 




92.5% 


92.1% 




100.0% 


75.9% 


Basic Skills Training 




25.5% 




41.3% 


21.1% 




0.0% 


16.7% 


Hard Skills Training 




73.4% 




67.5% 


71.9% 




81.8% 


83.3% 


Fixed Fee Contract 




91.9% 




93.8% 


92.1% 




81.8% 


90.7% 


Prior ETP Contracts 




12.0% 




8.8% 


9.6% 




18.2% 


20.4% 


Number of Amendments 




1.21 




1.34 


1.18 




0.64 


1.24 
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There were both expected and interesting variations in the characteristics of the contracts 
among the various groups. The group that deviated from the averages in the most 
interesting ways was the contracts with only administrative subcontractors. Particular 
areas of interest were in the size of the contract, the disencumbrance rate, the size and 
industry of the firms involved, and the type of training. 

The average size of a contract during the study period was about $300,000, with the 
original contract starting a little bit higher and the amended contract ending up a bit lower. 
Contracts with both training and administrative subcontractors followed the same pattern, 
and lost just a few thousand less in the amendment process. Contracts with only training 
subcontractors were about $75,000 smaller than average, and were concentrated among 
small and medium-sized businesses. In contrast, contracts with only administrative 
subcontractors were twice the average size — approaching $600,000 — and included only 
large employers. As one might expect, the contracts without subcontractors were about 
$85,000 larger than the average contract and were more concentrated among the large 
firms, which are more likely to have in-house training departments. This group was also 
twice as likely to have had a previous ETP training contract. 

The disencumbrance rate averaged about 40% for all groups except the one with only 
administrative subcontractors, which experienced a significantly larger 54% 
disencumbrance rate. It should be noted that 3 of the 1 1 contractors in this group had a 
100% disencumbrance rate and another had a 91% rate. The remaining 7 contractors had 
disencumbrance rates below 30%. This analysis of the individual contracts in this group 
suggests that any disencumbrance rate problem with this group is a problem of 
performance on selected contracts rather than a problem with the average contract in the 
group. 

The type of business served also varied by contract group. Given the average distribution 
of about 63% manufacturing, 30% services and 6% other, the group with both types of 
subcontractors served about 10 percentage points more manufacturers and 10 percentage 
points fewer service businesses than average, while the group with no subcontractors did 
the opposite. Again, the stand-out case was the administrative consultant only group 
which had about 63% of its contracts with service businesses rather than manufacturers. 

Less variation was observed in the type of training by contract group. However, 
administrative subcontractor only group had no basic skills training (present in about 25% 
of all contracts) and soft skills training in all of their contracts. This group also had more 
prevalence of hard skills training than average. The main type of hard skills training in this 
group were office skills training (6 contracts) and production training (5 contracts). 

Descriptive statistics of the sort presented in Table 2 may not reveal underlying 
relationships among all variables. To determine whether there were any underlying 
relationships that were not revealed in Table 2, we applied a logistical regression model to 
the data. The logistical model is especially suited for analysis of a choice between two 
alternatives. In this case, the choice is whether or not to subcontract with a training 
specialist or a contract administrator. We ran three logistical regressions with three 
different dependent (choice) variables - one with subcontracting in general (training 
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and/or administration) as the dependent variable, one with subcontracting training, and one 
with subcontracting administration. 

The results of the logistical regressions were not particularly surprising since they basically 
confirmed what was already obvious from the descriptive statistics. A larger contract 
increased the probability of it having an administrative subcontractor and slightly 
decreased the probability of it having a training subcontractor. Having a basic skills 
component increased the likelihood of the contract having an administrative subcontractor 
relative to just a training subcontractor or no subcontractor. One interesting result was 
that, relative to the Sacramento office, being administered out of the San Mateo or Los 
Angeles offices increased the probability of the contract having a subcontractor, 
particularly an administrative subcontractor. 

Overall, the regression analyses revealed that there was a high degree of variation with the 
identified subcontract and non-subcontract groups. This was manifest in the relatively low 
statistical significance indicators on the various regression coefficients. The results listed 
above were statistically significant at the 95% level, a few more were significant at the 
90% level (which is rarely reported). None of the regression results were significant at the 
99% level of confidence. 

Overview of Subcontractor Characteristics 

As we noted earlier, over the past three years, there have been approximately 300 
subcontractors and consultants involved in ETP contracts. Most have done only a limited 
amount of ETP work. In this section, we use the results of our survey of 34 subcontractors 
who were the primary subcontractors on an ETP project in the last year to provide a snap- 
shot of the subcontractor population. 

A Subcontractor and Consultant Typology 

While there is wide variety in the characteristics of people and companies who do ETP 
subcontracting and consulting, our survey data provides a profile of the typical primary 
ETP subcontractor. The typical primary subcontractor was in the training field before he 
or she began doing ETP subcontracting. They heard about ETP and started designing and 
delivering projects for their clients, and have been involved with ETP for 6.6 years. The 
typical subcontracting company is mid-sized with 6 to 20 professional employees, and 
provides a full range of ETP services: proposal development, administrative services, and 
training. ETP contracts make up about 40% of these companies’ revenue. 

Reviewing our data and what we learned about primary subcontractors, we found that we 
could classify these subcontractors and consultants into seven categories based on the 
characteristics of the organization and the services they provide. We believe this exercise 
is useful because understanding how subcontractors and consultants are organized, the 
services they provide, and the incentives that drive them can help to shape policy. Listed 
below are the categories we found and our estimate of the proportion of the subcontractor 
and consultant population that falls into each category. 



ERIC 



ETP and Its Subcontractors and Consultants 



24 



16 



• Full Service Companies (20%) : These companies range in size from 6 professional 
employees to nationally known management consulting firms with hundreds of 
professional employees. They provide a full range of ETP services including proposal 
development, administration, and a range of training programs. These firms can 
usually provide a complete training needs assessment and may own proprietary training 
packages which they customize for individual companies. They occasionally 
subcontract for specialized trainers or other services. 

• Public or Non-profit Agencies (5%): This category includes a wide range of public and 
private non-profit agencies. They range from small community based agencies, to 
unions, to large universities and community colleges. Some provide training only, 
others offer full service. We found their non-profit mission makes them a distinct 
group. 

• Hollow Companies ( 25% ): These companies market a full range of ETP services but 
are in fact small companies that subcontract out for most of the services they provide to 
a network of other small firms or individuals. Typically these companies do not 
employ full-time trainers and may have only one or two professional employees. Their 
primary expertise appears to be in developing proposals and administering projects. 
Most or all training is typically subcontracted out. 

• Niche Trainers (20%) : These companies are small highly specialized training 
companies, usually with few employees. They provide specialized training in narrow 
market niches. They may, for example, offer TQM training for medium-sized 
manufacturers or only provide vocational ESL. They may occasionally offer proposal 
development or administrative services for ETP contracts, in order to help a client 
afford a training package, but their basic business remains training. 

• Free-lance Project Administrators (20%) : This group is made up of individuals and 
occasionally partnerships, who primarily provide administrative services, often one 
project at time. They may occasionally help develop a proposal, but their assistance is 
primarily to get a proposal into ETP approved format rather than to design training. 

• Free-lance Packager (10%): This group is made up of individuals and occasionally 
partnerships whose primary business is seeking out companies who may be interested 
in ETP training, and helping them develop a proposal that will be accepted by the 
Panel. These companies generally do little systematic needs assessment. They may act 
as brokers for other companies that provide training or administrative services. 

• Project Doctor (<1%): Much like Hollywood script doctors, these subcontractors and 
consultants often become involved in a project after it has run into trouble! They tend 
to specialize in solving administrative problems and negotiating with ETP. This 
appears to be a very small group of subcontractors and consultants. 

It is important to note that individuals and companies may perform these different roles at 

different times as companies evolve over time. A free-lance packager may grow into a 



ETP and Its Subcontractors and Consultants 



17 



hollow company as the subcontractor or consultant becomes more experienced and builds a 
network of other companies. A niche trainer may become a full service company if it 
builds expertise in ETP and finds it profitable to provide a wider range of services. 

Details of Subcontractor Characteristics 

With this overview of subcontractors and consultants and how they operate, we now turn 
our attention to a more detailed examination of the subcontractors characteristics, as 
revealed in our project sample. 

Before individual subcontractors became involved with ETP, they were most likely to be 
working in the fields of training, education, management or management consulting. A 
complete breakdown of their previous fields is provided below. 

Subcontractor Background Before Working With ETP 



Training 
Education 
Management 
Management Consulting 

Engineering 

Human Resource 
Development 

ETP Employee 
Personnel 
Public Training Programs 
Union Official 
Other Government 
Other 

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 

Percent of Subcontractors 
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The survey information indicated that subcontractors became involved with ETP through a 
wide variety of experiences. The largest group included those already working as training 
subcontractors and consultants when they learned about ETP and started using it. Others 
were hired by a company that was already doing ETP work, and others were working for a 
company that did ETP work, and then decided to strike out on their own. The chart below 
shows the various routes subcontractors in our sample took into ETP work. 

Subcontractor’s experience with ETP ranged from 2-13 years with 6.6 years experience on 
average. 



How Subcontractors First Got Involved With ETP 



Was a training consultant 
and learned about ETP 

Got hired by current 
employer and got involved 

Worked for a firm that did 
ETP consulting 

Managed an ETP program 
as an employee 

Through meetings and 
publications 

Worked for ETP 
Other 




0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 



Percent of Subcontractors 



Another key characteristic of subcontractors is how dependent they are on ETP funds. We 
asked each subcontractor or company what portion of their revenue they received from 
ETP projects in the last year. The responses ranged from less than 1% to 100%. On 
average, the subcontractors reported that 40% of their revenue came from ETP. To analyze 
the data further, we broke the population into four groups based on the percent of revenue 
they received from ETP work. This seems important to us, because those contractors most 
dependent on ETP for revenue will be those most likely to be involved in attempting to 
shape ETP policy and those bringing the largest number of projects to the 
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Panel. As the table on the following page shows, we classified about 20% of the 
subcontractors as “ETP Dependent,” meaning that they got over 75% of their revenue from 
ETP projects; a second group of the same size was deemed “ETP Dominated,” because 
they got over half their revenue from ETP. About 60% got less than half their revenue 
from ETP. 



Table 3 

ETP Funds as A Percent of Total Revenue 



Group 

(Percent of Revenue from ETP) 


Percent of Surveyed Subcontractors 


ETP Dependent 

(75% or more of revenue from ETP) 


21% 


ETP Dominated 

(50% to 74% of revenue from ETP) 


21% 


ETP Active 

(20% to 49% of revenue from ETP) 


29% 


ETP Independent 

(<20% of revenue from ETP) 


29% 



When asked what other types of work the subcontractors did besides ETP funded projects, 
the most common responses were non-ETP funded training (73%) and management 
consulting (72%). 



How do employers choose subcontractors? 

It appears, from our survey data, that subcontractors tend to reach employers before they 
have heard about ETP. Forty-two percent of the employers surveyed said that they learned 
about ETP from their subcontractor. 
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How Employers First Heard About ETP Training 



A subcontractor 

One of their employees 

The CMA or other 
industry/professional group 

Worked with an ETP project 
before 

A newpaper or media story 

An ETP staff person or 
other ETP presentation 

The Dept, of Commerce or 
other state agency 

"Don’t know" 
Other 
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Percent of Employers 



Only half of the employers reported that they considered more than one subcontractor 
before hiring them. The employers that did “shop” for a subcontractor considered on 
average 2.8 subcontractors, indicating that most that did shop talked to two or three 
subcontractors. 
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one 



The largest group of employers reported that they found their subcontractor through a cold 
call from the subcontractor (31%). Twenty-two percent were referred to the subcontractor 
by another firm and 19% had used the subcontractor before. A complete breakdown of 
how employers found subcontractors is below. 

How Employers Found Subcontractors 



Contacted through a 
cold call & they used 

Referred by another firm 

Previous project or other 
consulting 

Special expertise & 
knew by reputation 

Former employee of the 
company 

Knew them when they 
were an ETP employee 

"Don't know" 
Other 
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When asked how they evaluated the subcontractor, slightly over half the employers said 
they based their evaluation on an interview with the subcontractor, slightly less than half 
reviewed the consultant’s literature. Surprisingly, only one-third called a previous client. 



How Employers Evaluated Subcontractors 
Before They Chose 



Persuasive presentation/ 
interview with subcontractor 

Reviewed literature provided 
by subcontractor 

Called former clients of the 
subcontractor 

Worked with subcontractor 
before & liked performance 

Referred by someone they 
trust 

Other 




0 % 10 % 20 % 30 % 40 % 50 % 60 % 70 % 80 % 90 % 

Percent of Employers 



In several interviews and focus groups, subcontractors suggested that ETP staff tried to 
discourage employers from using subcontractors. When we asked employers what advice 
they got from ETP staff about using subcontractors, 44% reported they got no advice and 
36% said the staff were neutral about using subcontractors. Only 13% reported the staff 
“said or implied” subcontractors were unnecessary, and 7% said the staff recommended 
using a subcontractor. 

How do subcontractors market their services? 

We also asked subcontractors how they generally got their clients. Overall, subcontractors 
reported that 20% of their clients came from referrals from non-ETP clients, and an 
additional 18% were referred by ETP clients. Fifteen percent of clients were existing 
clients for who the subcontractor had provided non-ETP services. Surprisingly, 
subcontractors reported that only 19% of clients came from cold calling or mailing and 6% 
from other business networking. 
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How Subcontractors Generally Obtain Their 

Clients 
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To gain some additional insight into marketing, we asked subcontractors how they got the 
particular employer sampled in our survey as a client. The majority (58%) said they found 
them through “other business networking,” about 20% had done other non-ETP funded 
consulting for the company and slightly over 10% said they obtained the client through 
cold calling. 



What services do subcontractors provide? 

We broke the ETP process down into its component parts and asked employers to tell us 
what tasks they primarily performed, what tasks the subcontractor primarily performed and 
which tasks were shared 8 . The following table shows how the work was divided for the 
projects in our sample, according to the employer. 

While there was wide variation in how work was divided in each project, our data allows 
us to draw a typical profile of the tasks done by employers and the tasks done by 
subcontractors. Overall, employers were primarily responsible for management tasks that 
directly involved the employees to be trained, financial tasks and providing facilities. 
Subcontractors were primarily responsible for dealing with the details of ETP compliance 
and delivering classroom training. 



8 Tasks were considered primarily performed by one party if they did two-thirds or more of the work. If 
neither part performed two-thirds of the work the task was considered shared. 
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Employers reported they typically were principally responsible for the following tasks: 



• training needs assessment, 

• initiate first contact with ETP staff, 

• develop budget, 

• appear in front of Panel, 

• design training, 

• recruit/identify participants, 

• intake participants, 

• provide classroom/lab facilities. 

• deliver SOST, 

• monitor SOST, 

• verify complete hours, 

• verify post-training retention, 

• supply information and meet with monitors, 

• invoice for completed participants, 

• follow-up for payment. 



Consultants typically were principally responsible for the following tasks: 



• write project narrative, 

• negotiate with ETP staff, 

• draft, revise and submit final proposal, 

• select and customize training materials, 

• deliver classroom/lab training. 
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Table 4 

How Work On ETP Projects Is Divided According To Employers 



Task 


Employer 


Subcontractor 


Shared 


Training needs assessment 


58% 


17% 


25% 


Write project narrative 


28% 


50% 


22% 


Initiate first contact with ETP staff 


49% 


43% 


8% 


Develop budget 


40% 


23% 


37% 


Negotiate with ETP staff 


34% 


43% 


23% 


Draft, revise and submit final proposal 


25% 


58% 


17% 


Appear in front of Panel 


41% 


28% 


31% 


Design training 


42% 


33% 


25% 


Select and customize training materials 


37% 


40% 


23% 


Recruit/Identify participants 


80% 


6% 


14% 


Intake of participants 


53% . 


24% 


23% 


Provide classroom and lab facilities and 
equipment 


92% 


6% 


2% 


Deliver classroom/lab training 


25% 


50% 


25% 


Deliver SOST 


52% 


37% 


11% 


Monitor SOST 


46% 


23% 


31% 


Verify completed hours 


50% 


31% 


19% 


Verify post-training retention 


69% 


20% 


11% 


Supply information and meet with monitors 
during project 


51% 


14% 


35% 


Invoice for completed participants 


41% 


41% 


18% 


Follow up for payment 


47% 


39% 


14% 



We asked subcontractors the same questions about the same set of projects. Not 
surprisingly the subcontractors reported they played a much larger role then the employers 
reported. According to the subcontractors, employers were typically principally 
responsible for the following tasks: 

• appear in front of Panel, 

• recruit/identify participants, 

• intake of participants, 

• provide classroom/lab facilities and equipment, 

• deliver SOST, 

• monitor SOST, 

• verify post-training retention, invoice for completed participants 9 , 

• follow up for payment. 



9 An equal number of subcontractor reported that this task was principally done by the subcontractor and 
employer. 
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According to subcontractors, they were typically responsible for the following tasks: 



• training needs assessment 10 , 

• write project narrative, 

• initiate first contact with ETP staff, 

• develop budget, 

• negotiate with ETP staff, 

• draft revise and submit final proposal, 

• design training, 

• select and customize training material, 

• deliver classroom/lab training, 

• verify completed hours, 

• invoice for completed participants. 



Unlike employers, subcontractors reported that some tasks were typically shared: 



• training needs assessment, 

• verify post training retention. 



Complete data on the subcontractor’s responses are presented in the following table. 



10 An equal number of subcontractors reported that this task was principally done by the subcontractor and 
principally shared. 
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Table 5 

How Work On ETP Projects Is Divided According To Subcontractors 



Task 


Employer 


Subcontractor 


Shared 


Training needs assessment 


12% 


44% 


44% 


Write project narrative 


19% 


62% 


19% 


Initiate first contact with ETP staff 


28% 


63% 


9% 


Develop budget 


19% 


43% 


38% 


Negotiate with ETP staff 


38% 


40% 


22% 


Draft, revise and submit final proposal 


19% 


47% 


34% 


Appear in front of Panel 


50% 


10% 


40% 


Design training 


13% 


50% 


r 37 % 1 


Select and customize training materials 


16% 


59% 


25% 


Recruit/Identify participants 


69% 


9% 


22% 


Intake of participants 


44% 


41% 


15% 


Provide classroom and lab facilities and 
equipment 


75% 


16% 


9% 


Deliver classroom/lab training 


15% 


70% 


15% 


Deliver SOST 


46% 


38% 


16% 


Monitor SOST 


42% 


39% 


19% 


Verify completed hours 


31% 


50% 


19% 


Verify post-training retention 


35% 


31% 


34% 


Supply information and meet with monitors 
during project 


36% 


30% 


34% 


Invoice for completed participants 


41% 


41% 


18% 


Follow up for payment 


48% 


39% 


13% 



In addition to asking about the tasks they performed, we asked subcontractors if they 
subcontracted any training or administrative work. Only 13% of the subcontractors 
reported they had subcontracted work on the projects we sampled. In all cases, the 
subcontracting was for training and not for administrative work. The average amount of 
training subcontracted was high, 70%. In several cases, 100% of the training was 
subcontracted. 

We also wanted to know if the subcontractors or the employer had the most direct contact 
with ETP. Fifty-four percent of the subcontractors reported that, on the projects studied, 
the employer had the most direct contract with ETP. We asked the 46% of the 
subcontractors who had most of the direct contact with ETP why they had most of the 
contact. The most common response was that the employers were simply too busy to deal 
with ETP, or that the employer considered it the subcontractor’s job to deal with ETP. 
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How satisfied are employers with the subcontractors they chose? 

Overall, most employers surveyed believed the subcontractors they hired played a key role 
in the success of their project. As the graph below indicates, over 80% of these employers 
said subcontractors were either “critically important” or “important” to the success of their 
project. Only 3% responded that the subcontractors did not help at all. 



How Important The Subcontractor Was To Project Success 



Critically important, could 
not have done it without 

Important, helped them 
substantially 

Helped but they could have 
done it themselves 

Did not help 

Undermined project 
performance 




Percent of Employers 



To further test employer’s satisfaction with the subcontractor they hired, we asked them if 
they would use the same subcontractor again, use a different subcontractor, or do the next 
ETP project on their own. Over two-thirds said they would use the same subcontractor 
again. We would classify these employers as “satisfied.” Twenty percent said they would 
do the project on their own. It is unclear whether these employers are satisfied or not. 
Finally, 1 1% said they would choose a different subcontractor if they did another ETP 
project. We would classify these employers as “dissatisfied.” 
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Would: 
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subcontractor 



Do the project on 
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Use a different 
subcontractor 
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These data persuade us that most employers remain satisfied with the subcontractors they 
choose. There does seem to be a problem group of subcontractors however. We estimate 
from this survey that between 10% and 20% of the subcontractors on the projects we 
studied did not perform up to the expectations of the employer. The reasons why this 
group of subcontractors did not meet the employer’s standards is unclear. We don’t know 
the degree to which the dissatisfaction is caused by miscommunication, a lack of 
competence on the part of the subcontractor, or other problems. 

Interestingly, when we cross-tabulated this evaluation of the subcontractor with whether or 
not the employer considered more than one subcontractor, we found that all the employers 
who were dissatisfied had not shopped for a subcontractor, but considered only one. 
Similarly, 3 1% of the employers who did not shop for a subcontractor said they would 
rather do the next project on their own compared to only 12% of those employers who did 
shop. All this indicates to us that a lack of information about alternatives does lead to less 
satisfaction with the choice of subcontractor. 

We attempted to answer this question by first asking the employers to identify all the 
benefits they got from using a subcontractor (see following table) and then to identify the 
“most significant” benefit. Clearly the most important benefit was in helping manage what 
the employers saw as the complex ETP process. Fifty-one percent of the employers gave a 
benefit related to managing the program as the most significant benefit received from using 
a subcontractor, while only 30% reported a benefit related to training. Fully one-quarter of 
the employers reported that help keeping track of complex ETP “rule and policy changes” 
was the most significant benefit. An additional 17% said “handling ETP paper work the 
employer’s staff could not,” and 11% believed a subcontractor “made it easier to get ETP 
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to approve the project.” Conversely, 19% said “provide trainers employer did not have in- 
house” and 1 1% reported “designed custom training” as the most significant benefit. 



Table 6 

Benefits That Companies Get From Using A Subcontractor 



Benefits 


Percent 

Response 


Most 

Beneficial 


Needed a subcontractor to keep track of all the 
complex rule and policy changes in the ETP 
program. 


67% 


25% 


Did not have enough in-house trainers. 


61% 


19% 


Needed subcontractors to design custom 
training. 


61% 


11% 


Believed it would be easier to get ETP to 
approve the project with a subcontractor. 


61% 


11% 


They handled ETP paperwork our staff could 
not. 


58% 


. 17% 


Did not have staff capable of managing 
program. 


44% 


6 % 


Cost less to use a subcontractor than to use staff. 


33% 


0 % 


Made it easier to get top management’s 
approval. 


31% 


0% 


Believed our company could get more money 
per participant with a subcontractor. 


25% 


0% 


Shared the risk of people not completing. 


14% 


0% 


Other 


25% 


11% 



Next we asked employers if the agreements they made with subcontractors about what 
tasks they would perform held up through the project. Over 80% of the employers reported 
that they had a clear agreement at the beginning of the project that held up throughout; 
fewer than 20% reported making significant changes in the agreement during the training. 
About 20% of the employers reported that their staff ended up doing work that the 
employers intended the subcontractor to do. In almost 90% of these cases, the employer, 
rather than ETP, paid for the additional work. 



How are subcontractors compensated? 

We found a wide range of compensation arrangements within the projects studied. For 
administrative subcontractors the most common arrangement is a fixed fee for the entire 
project. Surprisingly, few administrative subcontractors shared risk with the employer by 
receiving a fee just for successful completers. 



We asked the employers if the fixed fee was adequate to cover the direct costs of training. 
Direct costs were defined as cost of training exclusive of the salaries of trainees. Sixty-one 
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percent reported that the fee was adequate. We then asked the 39% that said the fixed fee 
for training was inadequate what percent of costs the fee covered. On average, employers 
estimated the fee covered 72% of their direct training costs. Estimates ranged from 50% to 
90%. 

Eleven percent of the employers reported that they ended up paying their administrative 
subcontractor more than the 15% administrative fee. While 19% of the employers reported 
paying the training subcontractor more than the fixed fee per trainee. We asked employers 
who said they paid additional money to their subcontractors why they did. The most 
common response given by 75% was that they had to pay money beyond the ETP fee to get 
the quality of training they wanted. Half said that the cost of training absorbed the 1 5% 
administrative fee, so administrative costs were paid out of company funds, and half said 
that the 15% administrative fee was used to pay the subcontractor for developing the 
proposal. One quarter reported that ETP funds paid the training subcontractor while the 
company paid in-house trainers. 

Subcontractors’ and Consultants’ Role in ETP 

Much of the controversy that swirls around ETP subcontractors and consultants deals with 
different group’s assessment of the role they play and the impact of that role. To some, 
subcontractors and consultants are opportunists looking for windfall profits who threaten 
ETP’s future. To others, subcontractors and consultants are white knights here to help 
make ETP work. Most observers see subcontractors and consultants as offering both 
threats and benefits to ETP. In this section, we review what we learned about different 
constituencies’ views of subcontractors and consultants and then offer our own assessment. 

Staff views of subcontractors and consultants: 

Many ETP managers and staff who were interviewed have a skeptical view of 
subcontractors and consultants. We believe such views are often shaped largely by the 
relatively few problem projects that come to the staff’s attention and take up a lot of their 
time, and which involve a subcontractor or consultant. This can be particularly true for 
managerial staff. They only see problem projects, not routine projects where 
subcontractors and consultants may have made a significant contribution to the project’s 
success. There appears to be a significant variation in the degree to which staff see 
subcontractors and consultants as a problem. Attitudes appear to vary somewhat from one 
regional office to another. 

Staff voiced several specific concerns regarding the role of subcontractors and consultants. 
Interviewed field staff are concerned that subcontractors and consultants “get to the 
employers first and demonize ETP to make the process seem horrendous.” Staff argue that 
subcontractors and consultants do this to convince the employer to hire them. This sets up 
an adversarial relationship between the employer and ETP. Some staff object that 
subcontractors and consultants keep them from having direct contact with senior managers, 
thus frustrating their efforts to make sure employers are aware of various issues. The 
example often given by staff is that they will call an employer and then get a call back from 



4 



ETP and Its Subcontractors and Consultants 



32 



the subcontractor or consultant. Field staff also worry that subcontractors and consultants 
who only work on ETP projects sporadically don’t keep current with ETP regulations and 
practices. As a result, they give employers bad advice and then blame field staff when 
there is a conflict. 

Staff also believe subcontractors and consultants have powerful financial incentives, 
especially if they are paid a commission when a project is approved - so called “fund and 
run” subcontractors and consultants, to pump up the size of projects beyond what the 
employer can reasonably expect to complete. This leads to high levels of disencumbrances 
which in turn cause a variety of problems for ETP. 

Staff charge that some subcontractors and consultants are offering low quality or 
uncustomized training and are hoodwinking unsophisticated employers into buying it 
because the money they are spending is the State’s. By offering only off-the-shelf curricula 
and low paid instructors, subcontractors and consultants can make excessive profits. 

Finally, some staff simply object to administrative subcontractors and consultants, because 
they believe they are unnecessary. These staff see the ETP process as simple and believe 
any employer can get through the process easily, and they simply do not need 
subcontractors and consultants. Most staff do acknowledge that many firms lack the 
capacity to provide training in-house and must hire training subcontractors and consultants 
to actually deliver training. 

Despite all these objections to subcontractors and consultants, when pressed, all field staff 
can mention subcontractors they have worked with who were well- informed and provided 
employers with valuable service. 

Management staff has some additional worries about subcontractors and consultants. 
Management staff often see subcontractors as constantly testing ETP policy, looking for 
loopholes and exceptions which allow them to get additional funding for their clients and 
ultimately for themselves. A related worry is that subcontractors and consultants seek out 
the largest employers because it is most profitable to serve them, thus skewing the mix of 
ETP projects towards larger companies. 

Quality of training is an additional issue for senior ETP staff. Senior staff is especially 
worried that the fixed fees have created an incentive for a group of marginal contractors to 
attempt to turn ETP into a cash cow by persuading employers to buy low quality, 
uncustomized, often irrelevant training. Getting these projects through the Panel for the 
existing fixed fee for training results in the reaping of large profits by keeping the cost and 
quality of training low. If such projects do exist and attract public attention, it will reflect 
badly on the program. 

Managers are also concerned that many subcontractors and consultants have “retreated to 
the shadows,” because they don’t want their agreements with employers scrutinized by the 
Panel. For the most part, managers think subcontractors and consultants are working hard 
to keep information about their practices and pricing scarce in order to keep the buyer in 
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the dark. In general, management staff is concerned that employers may not have the 
information they need to evaluate subcontractors and consultants, since there is little public 
information on their track record or fees. Management recognizes that many employers 
learn about ETP through subcontractors and consultants and that senior management in 
many companies have little direct contact with ETP. 

Senior managers believe that subcontractors and consultants attempt to manipulate ETP 
policy by going directly to the Legislature or the Panel and thus undermine the staff. 

A major issue for senior staff is the fear that aggressive marketing of the availability of 
ETP funds will bring in proposals for training that do not match ETP’s priorities. More 
specifically, managers believe subcontractors and consultants may seek out companies 
with existing training and simply get them to substitute ETP funding for existing funding, 
thus not adding at all to the amount of training done. 

Panel members ’ view of subcontractors and consultants: 

While Panel members hold a variety of views about subcontractors and consultants, 
overall, the Panel members we interviewed had a less negative and more balanced view of 
subcontractors and consultants than did the staff we interviewed. Most Panel members see 
subcontractors and consultants as legitimate businesses who contribute to ETP’s ability to 
create and execute projects. Panel members also recognize the downside of subcontractors 
and consultants and are particularly concerned about excessive fees. 

Panel members, who are principally involved with setting overall ETP policy, saw 
subcontractors and consultants as a particularly active group in the policy process. 
Individual members differed on how influential subcontractors and consultants actually 
were in the policy process. One member, for example, said subcontractors and consultants 
were as influential as major constituent groups such as labor unions, and the CMA, while 
another thought subcontractors and consultants, while very active, exercised little real 
influence. Several Panel members noted that subcontractors and consultants were quick to 
go directly to the State Legislature or to hire an attorney when they had a dispute with ETP. 

One perspective several Panel members had that was not found among senior staff, and 
that Panel members recognize, was that employers often need an advocate in the complex 
ETP process. In the words of one Panel member, “until we make the process less 
bureaucratic and more user friendly, companies will need subcontractors and consultants.” 

Several Panel members recognized that problems with subcontractors and consultants 
often spring from the fact that employers lack information about ETP in general and 
subcontractors’ and consultants’ performance and prices. Thus employers may be easily 
manipulated, especially when they believe they will get “free government money.” In 
general, Panel members were in favor of giving employers more information about 
subcontractors and consultants and their track records. 
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Subcontractors view of staff: 



There is a wide range of attitudes towards Panel staff among subcontractors. Many 
subcontractors, particularly those who have done only a few ETP projects, and those who 
are themselves former ETP staff people, see ETP staff as partners in the process and report 
good relations. Other subcontractors, particularly a group who are ETP dependent, see 
their relationship with ETP as adversarial. They view themselves as vigorous advocates 
for their clients in perpetual conflict with the staff. The subcontractors with the most 
negative view of ETP staff focus most on the work of monitors. In the words of one 
subcontractor, they get a “wall of hostility” from monitors. Another said monitors treat 
their clients “like they are on trial for a crime.” Several subcontractors agreed that they try 
to keep employers away from ETP; they use terms like “serving as a buffer” or “putting up 
the filter” to describe how they separate their client from ETP. While some subcontractors 
argued that they did not demonize ETP and attempt to frighten their clients away from 
working directly with ETP, others admitted they did “tell horror stories” about ETP, but 
only, they claimed, because they were true. 

Interestingly, most subcontractors we interviewed agreed that there is a problem group of 
subcontractors and consultants, many of whom are poorly informed about ETP, and who 
may make excessive profits and deliver little of value. They saw the problem as one of 
ETP staff being too concerned about the process and not caring enough about the quality of 
training that was actually delivered. In the words of one subcontractor, “ETP shouldn’t be 
concerned with who is making money, but with what clients (employers) are getting for the 
money.” 

Most of the subcontractors we interviewed supported the idea of ETP providing additional 
information about the experience and performance of subcontractors. There was less 
enthusiasm for revealing price information. 

Why Some Consultants Prefer to Remain Hidden 

We asked subcontractors in both the focus groups and the phone interviews why some 
consultants remained hidden and did not make official subcontracts for either 
administrative or training services. One subcontractor summed up hidden consultants 
quickly with the phrase “less liability, more flexibility.” 

To elaborate, one common reason why consultants remained hidden is that they did not 
want to share the risk of trainees not completing. The consultants believed that once they 
made a contract with the employer, if they provided the services they should be paid. 

Hence, if individual trainees failed to complete training, but the classes were offered, the 
training provider believes they should be paid in full. These trainers are quick to point out 
that they are not responsible for whether or not trainees complete, since the most common 
reason for not completing is that trainees are pulled out of training for production. 
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Several trainers also said that getting paid outside the contract guaranteed they would be 
paid promptly, if they had an official subcontract they would have to await ETP approval 
of payment. Other trainers said they charge substantially more than the fixed fee and did 
not want to sign a contract for a lower fee, even if they got supplemental payments outside 
the contract. 

Administrative consultants said that they remained hidden because development costs are 
not covered by ETP contracts and not ETP’s concern. Interestingly, the subcontractors 
were not uniformly opposed to ETP’s new policy of requiring that fees paid development 
consultants be revealed in the application. About half the subcontractors interviewed 
thought the policy would have no impact on them or other subcontractors and consultants. 
About 40% saw it having a beneficial impact say things such as: “It will help ETP monitor 
quality better,” or “It will help ETP keep track of subcontractors and consultants who 
cause problems.” Those who saw a negative impact, most often reported that it would 
discourage employers from applying for ETP money because they don’t want to see private 
transactions revealed in public. 

Impact of Subcontractors and Consultants on ETP 

In this section, we bring together all the information from the interviews, focus groups and 
surveys to summarize our assessment of the impact of subcontractors and consultants on 
ETP. Since subcontractors’ and consultants’ most severe critics will admit some 
subcontractors and consultants provide some valuable services, and since the most ardent 
defenders of subcontractors and consultants admit there are problem subcontractors and 
consultants out there, we have organized our summary around various aspects of the ETP 
program and attempted to demonstrate how subcontractors and consultants offer a benefit 
to ETP, and contrast this with the risks that they pose to the program. 

Marketing 

Benefits : This is the area with the strongest consensus. All the groups we interviewed 
recognized that subcontractors and consultants play a major role in marketing ETP. One 
senior manager estimates that they are responsible for bringing in as much as 80% of all 
ETP projects. Our data show subcontractors and consultants market aggressively to 
individual companies and reach hundreds of employers that ETP marketing could never 
reach directly. In general, it appears that subcontractors and consultants respond to the 
priorities and incentives created by the Panel. For example, when ETP made basic 
industries a priority, subcontractors and consultants shifted their marketing to those 
industries or left ETP consulting. To us this illustrates how ETP can use the powerful 
profit incentives that drive subcontractors and consultants to achieve its policy objectives. 

Risks : Subcontractors and consultants are aggressive, they will test the boundaries or push 
the envelope on ETP policies to try and qualify projects which are marginal, look for 
loopholes to slip through projects that may conform to the letter of ETP policy but not to 
the spirit of ETP priorities. A major risk we identified is that subcontractors and 
consultants may seek out large firms with substantial existing training programs and use 
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their knowledge of ETP to qualify these ongoing programs for ETP funding. 

Subcontractors and consultants may prefer this approach since it is less costly for them to 
simply package or repackage an existing program rather than go through the difficult steps 
of needs assessment and customizing training. The primary risk to ETP is that projects like 
this do not add to the stock of training but are simply substitutes for existing private 
training expenditures. 

In order to insure they are employed, subcontractors may demonize ETP and make the 
process appear more difficult then it actually is, undermining ETP’s own marketing efforts. 

Finally, subcontractors and consultants will not market to employers who cannot be served 
profitably, whether they are an ETP priority or not. Such is the case with small employers 
who subcontractors argue can not be served profitably under a fixed fee contract. 

Situations like these undermine ETP’s ability to achieve its objectives if it relies on 
subcontractors to bring in projects. 

Developing Projects 

Benefits : All the groups we interviewed recognized that many of the training and 
management companies that do ETP subcontracting have a wealth of specialized expertise 
in skill areas, in industries, and managing projects. Good subcontractors and consultants 
help companies do effective needs assessment that links training to larger corporate goals, 
they deliver high quality state-of-the-art training that is customized for the individual 
companies. They help companies design programs that are appropriate in size and scope 
and that disrupt productions as little as possible. Subcontractors also bring innovation to 
ETP in the design of projects and the delivery of training, such as the development of 
consortia to serve small businesses. 

Finally, knowledgeable subcontractors and consultants will sort out projects and keep 
projects which are ineligible from coming to the staff. Also, by developing proposals 
which conform to ETP standards, subcontractors and consultants save both staff and 
employer time. 

Risks : Some subcontractors and consultants driven by powerful profit incentives will try 
to shape projects to be most profitable for them. This may mean creating projects which 
are larger then they should be, substituting off-the-shelf curriculum for customized training 
or not conducting a careful needs assessment to insure training will lead to desired 
company goals. Subcontractors and consultants will sometimes evade or manipulate rules 
to increase profits. Subcontractors and consultants may not always act in the best long- 
term interest of an employer, they may, for example, encourage employers to design larger 
projects than optimal for success under an ETP project. 

Managing Projects 

Benefits : Effective subcontractors and consultants can save monitoring staff time by 
knowing ETP procedures and providing needed documentation and information promptly. 
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Subcontractors and consultants help implement ETP policy changes by tracking changes 
and keeping employers up-to-date. Subcontractors and consultants can specialize and 
develop expertise in industries or types of companies, such as unionized companies, and 
become experts in the specialized issues that may apply to these companies, again 
potentially smoothing the way for project management. 

Subcontractors have developed specialized software for tracking hours and other ETP 
related tasks, which again improves the efficiency of project administration. 

Risks : Subcontractors and consultants may distort ETP policies and undermine 
effectiveness, by testing boundaries, seeking loopholes in order to serve existing clients or 
generate more profits. For example, through a tortured rationale, a subcontractor or 
consultant may force training into an ETP category where it does not really belong in order 
to get it funded. Subcontractors may deliberately limit contact between the employer and 
ETP staff in order to maximize their own value. This undermines ETP’s ability to 
communicate with top management and insure that they understand ETP. 

Finally, subcontractors and consultants may simply have wrong or old information. Many 
subcontractors and consultants do ETP projects only sporadically and may not keep up 
with changes in ETP policies, hence they may give employers bad information which will 
cause conflicts between the employer and project monitor later. 

Delivering Training 

Benefits : Today, even relatively large companies may have few or no in-house trainers. 
Training subcontractors and consultants can provide ETP programs with high quality state- 
of-the-art curricula, and trainers. Without training subcontractors and consultants, 
companies without an in-house training capacity could simply not participate in ETP. The 
competitive training market generates innovation in training as training companies 
compete with each other. This process saves ETP the cost of developing curricula and 
training trainers, all of which are costs borne by traditional training agencies such as the 
community colleges. Subcontractors and consultants have industry expertise which ETP 
staff could not develop. 

Risks : Training subcontractors and consultants have an incentive to reduce the quality of 
training and hence its costs in order to reap larger profits. For example, a training 
subcontractor or consultant could substitute a less experienced instructor for a more 
experienced one, or standard curriculum for a customized one to save costs. This is 
particularly a problem if the subcontractor or consultant does not expect repeat business , 
from the company. 

Policy Process 

Benefits : Subcontractors and consultants spend a lot of time working with employers and 
the ETP process, hence they have extensive and detailed knowledge of how ETP policies 
are operationalized in the field. Tapping this understanding, as this project tried to do, 
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could provide the Panel with valuable information for policy development. Also, to the 
degree that subcontractors are representing the view of employers, they bring that 
perspective to the development of policy. As organizations and individuals who are 
dependent on ETP, at least in part for their living, they have some incentive to see ETP 
succeed and hence have effective policies. 

Risks : Once again, as profit driven entities, subcontractors and consultants are likely to try 
to manipulate the policy process solely for their own gain. Also, they may serve as 
advocates for a narrow group of employers, and overlook broader issues which the Panel 
must consider. 

Recommendations 

It seems to us, that given the current structure of ETP, subcontractors and consultants will 
remain a part of the program. The challenge for the Panel is to find policies that will 
enhance the benefits provided by subcontractors and consultants and to minimize the risks 
to ETP posed by their involvement. 

The Panel is confronted with a continuum of policy options that range from laissez faire to 
aggressive intervention, with a host of options in between. 

A Continuum of Policy Options 



Laissez Faire 


More Market Information 


Licensing 


l 


1 


1 


Least Intervention 




Most Intervention 



At one end of the continuum, the Panel could take a laissez faire approach. They could 
even do away with the existing regulations for subcontracts. At the other end of the 
continuum, the Panel could attempt to evaluate the qualifications of subcontractors and 
license approved subcontractors. In this case, only approved subcontractors would be 
allowed to work on ETP projects. 

Given the substantial amount of public funds involved in ETP projects and the problems 
that have occurred with subcontractors and consultants, we cannot in good conscience 
recommend a laissez fair approach. On the other hand, given the complexities and legal 
problems of issuing licenses, and the reduction in competition caused by licensing, we 
cannot recommend such an extreme intervention. 

Rather, we recommend a middle course. Our recommendations have two overarching 
goals: (1) to align the incentives that drive subcontractors and consultants with the larger 
objectives of the Panel and (2) to transform the inefficient market for ETP subcontractors 
and consultants into an orderly market. To accomplish these goals, we recommend that the 
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Panel act to increase the amount of information available about subcontractors and 
consultants, and take other steps to help employers make informed choices about 
subcontractors and consultants. In our view, increased information will make employers 
better consumers of subcontractor or consultant services, which should in-tum drive down 
the price of subcontractor or consultant services and improve the overall quality of 
subcontractors and consultants. Finally, all of this can be accomplished at a modest cost. 
Detailed recommendations and elaborated arguments for this strategy follow. 

1. The Panel should provide employers with an easily accessible system of 
consumer information on subcontractors. 

Our data show that both sellers (subcontractors and consultants) and buyers (employers) 
are operating in a inefficient market with little information. For a variety of reasons, 
employers are not shopping extensively for subcontractors and consultants. In fact, half 
the employers report they spoke to only one subcontractor before employing them. Despite 
the fact that most employers are satisfied with the subcontractors they chose, there is a 
substantial minority, probably between ten and twenty percent, who are dissatisfied with 
the subcontractors or consultants they hired and this group of employers tended to shop 
less than more satisfied employers. 

As we noted in the beginning of the report, government often intervenes in inefficient 
markets to increase the amount and availability of information so that the market can 
function more efficiently. That is the course of action we recommend here. 

Both employers and subcontractors viewed an information system as a valuable and 
legitimate activity for the Panel to undertake. In our survey, we asked both employers and 
subcontractors what they thought of the idea. Ninety percent of employers reported that 
information about subcontractors supplied by ETP would be valuable to them. Seventy 
percent of subcontractors report that such an system would be a “a good idea.” Overall, 
both parties believed the situation would be improved by the availability of objective 
information about subcontractors. 

We recommend the system focus on collecting data about subcontractors. Since most 
subcontractors also work as consultants at other times, the system will automatically 
provide data about most consultants. 

We note that recently ETP staff have produced a spreadsheet with information about 
subcontractors who have worked on ETP projects in recent fiscal years. This is an 
excellent first step towards creating a consumer information system. 

Information to be included in the system 

We recommend that the system include the following types of information: 

• Name and contact information for all subcontractors who have been listed as 
subcontractors on ETP projects in the last three years. 
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• The names and contact information of ETP clients each subcontractor has had 
over the last three years. 

• Descriptive information of the subcontractor’s work on the projects, including: 
project size, industry, type of training or administrative services delivered. 

• Information on the fees paid to subcontractors for services provided including: 
proposal development, administration, and training. 

• Percent of planned trainees who completed training on each project. 

• A rating of the subcontractor’s performance by the employer. 

• Subcontractor’s attendance at recent ETP training sessions. 

In our survey, we asked employers and subcontractors to rate how valuable they thought 
each type of data would be. The results show a surprising consensus. The chart below 
shows the percent of subcontractors and employers rating each item as very valuable or 
valuable. 



Employers & Subcontractors Who Consider Information 
To Be Valuable Or Very Valuable 



Names and contact info, for 
ETP clients 




89% 



Rating of subcontractor 
performance 




89% 



Size and scope of ETP 
projects 




■ Employers 
□ Subcontractor 



% of planned trainees 
completed 




80% 
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The names and contact information of former ETP clients was rated highest by both 
groups. This basic information would allow employers to quickly contact former clients to 
get information on the subcontractors’ performance and allow the employer to see how 
many clients the subcontractor has served in the last three years. 

Information on the size and scope of previous ETP projects was rated as valuable by over 
80% of the employers, but just 70% of subcontractors. Employers appear to be interested 
in this data, because they could see if the subcontractor had worked for employers similar 
to them. These data are readily available in existing ETP databases. 

The percent of planned trainees completed was a more controversial item. Eighty percent 
of employers rated it as valuable, they saw it as a key indicator of subcontractor 
performance. In contrast, only 60% of subcontractors saw it as valuable. A number of 
subcontractors argued that the number of trainees completing was out of their control, as it 
is the employer’s decision whether or not to release people to complete training. In our 
view, subcontractors are not solely responsible for planned trainees not completing, but in 
orienting employers to the ETP system, conducting needs assessments, and planning and 
administering projects, subcontractors do have a role in determining the proportion of 
trainees which will complete. 

Employer’s ratings of the subcontractors’ performance was viewed as valuable or very 
valuable by almost 90% of the employers and 67% of the subcontractors. Typically, 
employers wanted to see answers to the types of evaluative questions they would ask a 
fellow employer if they called to check on a subcontractor’s earlier performance. Some 
subcontractors resisted this item because they believed that if an employer was dissatisfied 
with the outcome of their ETP project, or frustrated with ETP’s monitoring, they would 
blame the subcontractor and give them low ratings. 

We asked both employers and subcontractors what aspects of a subcontractor’s 
performance should be rated. Based on their response, interviews with ETP staff and Panel 
members, and our own experience with ETP projects, we suggest subcontractors’ 
performance be rated in the following categories." 

Administration 



• Knowledge of ETP policy and procedures in the proposal process. 

• Knowledge of ETP policy and procedures in project administration. 

• Reliability and professionalism: availability to employer, ethics, ability 
to work effectively with ETP staff. 

Training Services 

• Quality of training curriculum: sequence of material, appropriateness of 
the level of material, appropriate weighting of material. 



11 It is important to note that subcontractors will only be rated in areas where they provided services. 
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• Degree to which training was customized to the employer’s needs. 

• Performance of classroom trainers. 

• Quality of SOST supervision. 

Overall 

• Degree to which training achieved expected results. 

• If employer would use the subcontractor again for a similar project. 

Collecting the Information 

Except for the ratings of performance, the information we propose for the system is already 
collected in the ETP project file, it is just a matter of assembling it into an easy to 
understand format. 

The rating information is new information. Both employers and subcontractors agreed that 
if such information is collected it should be collected by a third party, not ETP staff. We 
would recommend that whenever a project with a subcontractor closes, a third party under 
contract to ETP phone the employer and complete a brief structured interview to collect the 
ratings. It is important the individual collecting the ratings data talk to the most 
knowledgeable person on the employer’s staff. In some cases, the interviewer may have to 
speak to more than one person to get complete ratings. To protect employers and insure 
that ratings are as frank as possible, we recommend that ratings not be posted by employers 
but rather that summary ratings be posted for subcontractors. 

To insure a quick start-up of the system and reduce costs, we recommend that the system 
be phased in. In year one, three years of contact information and descriptive data from 
existing ETP records could be posted, but ratings data would only be collected for projects 
that closed in the last year. In each subsequent year, a new year’s worth of ratings data 
would be added until three complete years of data were available. 

Distributing the Information 

The proposed system will only succeed if the data are put directly into the hands of 
employers as they’re considering applying for an ETP project. To accomplish this, we 
recommend that the data be available on the ETP Web-site and that the availability of the 
data be broadly publicized in all ETP marketing efforts. In addition, we recommend that a 
paper copy of the data be included with each application package that is sent out to insure 
that at an early stage employers have the information they need to choose a subcontractor 
wisely. This is critical, since our data show 40% of the employers learned about ETP from 
a subcontractor or consultant, so many employers are in contact with a subcontractor or 
consultant from the very beginning of the process. The information may not be used to 
select a subcontractor in these cases but it will provide information to help the employer 
negotiate the best price they can. 
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2. 



The Panel should develop a consumer guide to help employers choose 
subcontractors and consultants wisely. 

ETP should develop a brief readable guide that will explain to employers why they may 
want to use or not use a subcontractor. If they choose to use a subcontractor, it should e p 
them choose wisely by directing them to the new information system, suggesting that they 
consider more than one. It should also provide them with a list of questions to ask 
subcontractors before they hire them. Again, such a guide could be disseminated widely as 
part of ETP marketing efforts and be included with each application package. 

In addition, the guide should help employers understand the real costs of ETP training. In 
other words, explain that ETP funding is not “free money.” Rather the guide can 
emphasize that employers should spend the money as if it were their own. Illustrative 
budgets could quickly show employers the costs they will have to bear. The trade-offs 
between hiring a subcontractor to perform different tasks and keeping those tasks in-house 

could also be illustrated. 

3. The Panel should provide semi-annual training for subcontractors and post 
attendance on information system. 

To increase the level of information about ETP’s shifting policies and practices, ETP 
should provide half-day training sessions once or twice a year, at each ETP local office, to 
keep subcontractors up-to-date about changes in ETP policies and practices. The training 
sessions can cover recent changes in ETP policy and day-to-day operations. It will give 
subcontractors a chance to ask questions outside the context of a dispute over a particular 
project and give ETP staff an opportunity to explain and clarify policies. 

Attendance should be voluntary, but to provide an incentive for subcontractors to 
participate, ETP should post on the information system whether or not a company or 
individual attended ETP training. In the case of a company, the name and title of all the 
individuals who attended can be listed as well. This will provide an indicator to employers 
about currency of the subcontractor’s knowledge and provide a regular method for 
improving communication between local offices and subcontractors. 

4. The Panel should work with subcontractors to get advice and insights about 
issues related to ETP. 

Our research clearly demonstrates that subcontractors play a major role in implementing 
ETP Many subcontractors have developed substantial expertise in ETP and have a keen 
knowledge of how ETP policies and practices will affect employers in specific industries. 
We recommend that ETP tap this expertise by creating a “user group” of subcontractors 
from whom ETP can regularly seek advice and consultation. We are not suggesting that 
subcontractors be given a special role in shaping policy, but rather to use subcontractors to 
get feedback on the likely consequences of policy decisions that have been made on 
employers, and detailed suggestions for implementation. 
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Experience shows it is difficult to get busy employers to participate in these types of 
groups. Working with subcontractors will, to a degree, provide a method of tapping 
employer opinion, as subcontractors can speak for them in some situations. 

These groups will also provide a channel to improve communication between staff and 
subcontractors and perhaps reduce the distrust which has come to characterize many of the 
interactions between subcontractors and staff. 

5. The Panel should consider how subcontractors and consultants shape the 
consequences of ETP policies, and conversely how ETP policies drive 
subcontractor and consultant behavior when developing policies. 

All public policies have the potential to generate unintended consequences which 
undermine the initial intent of the policy. For example, several years ago to reduce 
spending, ETP created a policy which required that if an employer applied for a second 
ETP project they would receive 30% less per trainee than in a first- time proposal. 
Responding to the incentives created by this policy, subcontractors advised the employers 
they worked with to increase the size of their projects to make sure every worker who 
might need training was covered. This led to an increase in the disencumbrance rate and a 
host of related problems. 

Many other policy changes could reshape incentives for subcontractors and consultants 
thus changing their behavior. Since a large number of employers get their information 
about ETP from subcontractors and consultants, it may change their behavior as well in 
unintended ways. In developing policies, staff and the Panel should use their 
understanding of subcontractors and consultants to reason through the likely consequences 
of policies for subcontractors and consultants and shape policies accordingly. 
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Appendix A: 

Estimates of The ETP Training Market 

and 

The ETP Subcontractor and Consultant Market 
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The ETP Created Market for Training 



ETP has expanded the market for training in California through its payroll-based tax and 
subsequent expenditure of those monies on training. Because these funds are released 
through training contracts with employers or other principals rather than through state- 
provided training, the ETP expenditures constitute an addition to the market for private 
training in California. Private training can be provided though training specialists either 
employed by the ETP contractor or through independent training specialists who contract 
with the principal to provide the training. In this way, the ETP training funds create or 
expand the market for private training specialists in California. The presence of ETP 
expands the market both for people who directly deliver training (of the type permitted and 
encouraged by ETP) and for people who administer training projects. While this study 
focuses on independent subcontractors who contract to deliver ETP funded training and 
administrative services, it is important to remember that a training program that is 
implemented completely by the employer’s staff still requires training and contract 
administration services and therefore expands the market for private training delivery and 
administration in California. 

In this estimate of the expansion of the market for private training professionals created by 
ETP funding our analysis includes all of the 394 ETP training contracts during the 1994-95 
and 1995-96 fiscal years. The contract database for this market impact analysis is 394 
contracts rather than the 260 contracts used to develop the descriptive statistics presented 
in Table 2 in the text. 

ETP approved $145,140,000 in the 394 original training contracts during the 2 study years 
and, initially, that would seem to be a good estimate of the expansion of the private 
training market in California due to the presence of ETP. However, the training market 
impact estimate is complicated by several factors. The contract amount was amended 
downward during the course of the contracts to $136,725,000, and the amount actually 
earned by the contractors during the 2 year period was only $74,985,000, about half the 
amount originally approved. The true impact of ETP on the training market probably lies 
in the low end of the broad range between the $74,985,000 in ETP funds actually earned by 
contractors and the $ 1 36,725,000 in the amended contract amounts. 

There are several considerations that influence the likely level of the impact of ETP within 
this range. Beginning at the low end of the range, factors that suggest a higher estimate 
include: 



• ETP contract funds are disencumbered because the proposed training did not even 
begin in most cases, but about 20% of the disencumbered funds are associated with 
training that begins but is not completed for some reason (according to staff estimates). 
To the extent that some training takes place even though the funds are disencumbered, 
the ETP’s estimated impact on training will be higher. 

• About 40% of the ETP contractors surveyed said that ETP funds covered only about 
72% of their direct costs of training (which do not include the wages of the trainees) 
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because they wanted a higher quality of training than the fixed fee would allow. Fixed 
fees were used in contracts that accounted for 80% of the dollar volume of earnings. 
This suggests that ETP’s impact on training will be higher than their direct expenditure 
on training. 

A couple factors argue for a lower level of ETP impact than the top end of the range: 

• The amendments that reduce the original level of the contracts and the 
disencumbrances from the contracts are used to fund new contracts. If all of the 
original contract amounts were spent on training by ETP, less would be available for 
future contracts. That is, if no monies had been amended out of contracts or 
disencumbered, ETP could not have written over $135 million in original training 
contracts during the 2 year period because they did not have access to that much 
money. 

• To the extent that some of the training financed by ETP would occur even if ETP did 
not exist, the impact of ETP on total training is less. This would be the case even if a 
project that was financed by ETP would only have been privately undertaken at a much 
reduced level and scope. In this case, ETP still expands the private training market but 
only by the amount of the difference between the value of the training that would have 
occurred without ETP and that which did occur in the presence of ETP. 

The large estimated range of ETP’s impact on the training market is not very satisfying and 
argues for some refinement of the estimate. To develop a more precise estimate, we begin 
with the amount actually earned in the contracts and then adjust for most of the factors 
listed above. 

The amount earned in ETP contracts: $74,985,000 

Adjustment for 20% of disencumbered funds actually involving training, 
assuming the training is 50% completed on average: $6,175,000 

Adjustment for ETP fixed fee under-funding desired training, 

40% of surveyed contractors reported 72% funding of direct training costs 

applied to fixed fee contracts which were 80% of the total 

dollar volume: $5,150,000 



Estimated 2 year impact of ETP on Direct Training Market: $86,310,000 



This estimate does not include an adjustment which nets out the value of the training that 
would have been done in the absence of ETP. For example, if 20% of ETP funded training 
would have been done in the absence of ETP, the estimate of ETP impact on training 
would be reduced to less than $70 million. Such an estimate is far beyond the scope of this 
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study. However, ETP can help maximize its overall impact on training in California by 
vigilantly avoiding paying for training that would have occurred in any case. 



The Market for Subcontractors and Consultants 

The focus of this part of the study is the extent and nature of the market for independent 
training and administrative subcontractors and consultants spawned by ETP contracts. 
Unfortunately, the overall extent of the market for subcontractors and consultants created 
by ETP funding is not immediately apparent from the available ETP data for two reasons. 

First of all, ETP collects contract information which includes only formal subcontracts for 
training and administration of the project, which we call subcontractors. It is well known 
by ETP staff and others that there have been consultants working with the contractors who 
have not shown up in ETP contract information; these are the consultants and their share of 
the market is difficult to estimate. Secondly, the ETP subcontract data are not adjusted for 
subsequent disencumbrances. An estimate of the market for subcontractors and 
consultants requires an adjustment procedure similar to the one developed for the ETP 
training market impact. 

In the following sections, the administrative and training subcontractor markets are 
estimated based on the original contract information; then the administrative and training 
consultant markets are estimated on the same original contract basis and data from the 
employer survey. Finally, the subcontractor and consultant market estimates are adjusted 
to reflect disencumbrances and underpayment in fixed fee contracts. 

The Market for Subcontractors (not adjusted for disencumbrances and fixed fee contracts). 

Our analysis of the 394 contracts yielded the following facts: 

• A total of 272 contracts (or 69.0%) of the 394 ETP contracts had reported training 
and/or administrative subcontractors. 

• These 272 contracts with subcontractors accounted for $79,925,000 (or 58.5%) of the 
$ 1 36,725,000 in total amended ETP contracts over the two-year study period. 

• The dollar amount of the subcontracts with reported consultants and subcontractors in 
these ETP contracts was $41,095,000, which was 51.4% of the allocation in these 272 
contracts, and 30.1% of ETP’s total contract allocation over this period. 

• Training subcontracts were found in 25 1 of the 272 contracts with subcontractors and 
totaled $36,535,000 or 52.9% of the average contract that had training subcontractors. 

• Administrative subcontracts were found in 127 of the 272 contracts with 
subcontractors, and totaled only $4,560,000 or 10.0% of the average contract that had 
administrative subcontractors. 
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• Training subcontractors clearly dominated the market both in terms of prevalence and 
dollars funded. Of the 272 contracts with subcontractors, 251 reported training 
subcontracts. The total dollar allocation on training subcontractors was $36,535,000, 
which was 88.9% of the allocation to subcontractors and 30. 1% of total ETP original 
contract amount over the period. Administrative subcontractors accounted for 
$4,560,000, which was 1 1 . 1% of the subcontractor allocation and just 3.3% of the total 
ETP approved contract funding over the period. 

On the basis of the amended contract funding, the potential market for subcontractors is: 

Training subcontractors: 

Administrative subcontractors: 

Total subcontractor market: $41 ,095,000 



$36,535,000 

4,560,000 



The Market for Consultants (not adjusted for disencumbrances and fixed fee contracts). 

We derived the estimate of consultants from the survey of 1995-96 contractors who did not 
report any subcontracts (subcontractors) in their original contracts or subsequent 
amendments. We interviewed 37 contractors out of the 38 contracts that showed no 
subcontractors to determine whether they had hired consultants who were not reported in 
the ETP contract. The 38 contracts in 1995-96 can be viewed as a random sample of the 
1 22 contracts without reported subcontractors during the two year period. Selecting recent 
contracts should not introduce any systematic bias into the sample and it certainly made it 
easier to gather specific information about the contract. The respondents indicated that: 

• Some type of consultant was used by 35.1% (13 out of 37) of the contractors. 

• Training consultants were used by 24.3% (9 out of 37) of the contractors. 

• Consultants to help develop the ETP proposal were used by 2 1 .6% (8 out of 37) of the 
contractors. 

• Administrative consultants were used by 10.8% (4 out of 37) of the contractors. 

• Administrative and/or development consultants were used by 24.3% (9 out of 37) of 
the contractors. 

• A person with ETP contract experience was actually hired on staff in 2 contracts to 
implement the training contracts. 

For estimation purposes we grouped the consultants who helped develop the proposals 
with the administrative consultants because information gathered in the focus groups and 
the subcontractors’ survey suggested that a common responsibility of an administrative 
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consultant is the development of the contract proposal. We then applied these percentages 
to the 122 contracts in the 2 year period that did not report using subcontractors. The two 
contracts under which an ETP-experienced professional was hired on staff were not 
counted in this estimation. The final estimates below would be higher if the hired 
professionals were considered to be “consultants” and included in the estimates. 

The first step in the estimating procedure is to determine the likely number of contracts 
during the period that had consultants. This entails applying the sample percentages to the 
population of 122 contracts without reported subcontractors. 

Estimate of the number of contracts with training consultants: 

(24.3%) X (122 contracts) = 30 contracts 

Estimate of the number of contracts with administrative consultants: 

(24.3%) X (122 contracts) = 30 contracts 

Estimate of the total number of contracts with consultants: 

(35.1%) X (122 contracts) = 44 contracts 

Our analysis of the specific contracts that reported consultants indicated that the average 
size of these contracts was substantially smaller than that of the group of contracts without 
reported subcontractors. The average size of those with consultants was $295,000 or 
63.3% of the overall average for the group without reported subcontractors ($466,000). 

This means that the likely dollar amount paid to consultants was smaller than the estimated 
proportion of contracts with consultants would suggest. 

This smaller size for the contracts involving consultants relative to the average contract is 
consistent with the results among the reported subcontractors versus the no-subcontractor 
group. The average size of a contract with subcontractors was 71 .4% of the average size of 
contracts without subcontractors. 

We assumed that the consultant contracts had the same proportion of the original contract 
amount designated for consultants as did the subcontractor contracts; that is 52.9% for 
training and 10.0% for administrative consultants and subcontractors. We accounted for 
the smaller average size of contracts with consultants by using the average size of 
consultant contracts ($295,000) in the calculation. These assumptions yielded the 
following estimates for training and administrative consultants: 

Estimated training subcontract dollar amounts: 

($295,000) X (52.9%) X (30 contracts) = $4,682,000 

Estimated administrative subcontract dollar amounts: 

($295,000) X (10.0%) X (30 contracts) = $885,000 
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Estimated Total Consultants Market during fiscal 1994-95 and 1995-96: 



Subcontractor 



Consultant 



Total: 



Training: 

Administrative: 

Total: 



$36,535,000 

4.560.000 

$41,095,000 



$4,682,000 

885.000 

$5,567,000 



$41,217,000 

5.445.000 

$46,662,000 



At this stage, the total estimated consultant payment amount is $ 5,567,000. As explained 
above, this amount does not reflect adjustments to the contract amount through the 
amendment process or due to disencumbrances. Nor does it include adjustments for the 
underpayment of direct training costs reported by 40% of contractors who claimed ETP 
contract payments only covered 72% of their direct training costs. These adjustment are 
made in the next section. 

The Overall Market for Subcontractors and Consultants: Adjusted for Disencumbrances 
and Fixed Fee Under-funding. 

Our estimate of the post-disencumbrance market for consultants and subcontractors takes 
the same basic approach as our estimate of the ETP-created market for training, but we 
factor in some unique features of consultant-contractor relationships. We beg in with the 
estimate of the total dollar amount in approved contracts that was earmarked for either 
consultants and subcontractors, then discounted by the average disencumbrance rate for the 
contracts in the sample. The result of that calculation is adjusted to account for training 
that actually occurred but for which the funds were disencumbered anyway. The final 
estimate of the market for consultants and subcontractors for ETP contracts is a range 
based on alternative assumptions regarding consultant-contractor payment agreements. 

The average disencumbrance rate in contracts with subcontractors at 5 1 .4% was virtually 
identical to that for the sample contracts with consultants (5 1 .6%). We used the 5 1 .4% 
discount factor since it applied to many more contracts. Applying the average discount 
factor of 5 1 .4% yields: 



As before, we assume that training actually occurs in 20% of these cases where funds were 
disencumbered, only in this case we assume that the training and administrative 
consultants and subcontractors get paid in full for the portion of training that occurs: This 
calculation adds back in 20% of the estimated disencumbrances in the preceding 
calculation. 



Training contracts: $41,217,000 X 48.6% = $20,031,000 

Administrative contracts: $5,445,000 X 48.6% = 2,646,000 

Fully discounted total consultant estimates: $22,677,000 
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In this estimation, we also adjust for the under-funding of direct training costs reported by 
40% of the contractors who were under fixed fee contracts. These 40% reported that the 
fixed fee covered only 72% of their direct costs of training. Training subcontracts with 
fixed fees accounted for 87.7% of the dollar volume of training subcontracts and 
administrative subcontracts with fixed fees accounted for 81.7% of the dollar volume of 
administrative subcontracts. These percentages yield an additional factor of 13.7% for the 
training consultants and subcontractors and 12.7% for administrative consultants and 
subcontractors. 

Adding back in the 20% of disencumbrances and the fixed fee adjustment factors yields 
our minimum estimate of the subcontractor and consultant market. 

Training contracts: 

adjustment for disencumbrance: $20,031,000 + (.2 X $21,186,000) = $24,268,000 
+ adjustment for underpayment: $24,268,000+ 13.7% = $27,593,000 

Administrative contracts: 

adjustment for disencumbrance: $2,646,000 + (.2 X $2,799,000) = $3,206,000 

+ adjustment for underpayment: $3,206,000+ 12.7% = $3,613,000 

Minimum estimated subcontractor and consultant market 

for the two-year study period: $31,206,000 

Information from our subcontractor focus groups and our surveys of subcontractors and 
employers led us to believe that subcontractors and consultants are actually paid more that 
the preceding calculations suggests. Our conversations in the focus groups with 
subcontractors suggested that very little risk-sharing was undertaken by subcontractors and 
consultants, particularly training subcontractors and consultants. Trainers were paid the 
contract amount even if the classes were only half full. Following this logic, we calculated 
an upper estimate of actual subcontractor payments based on the assumption that 
subcontractors were paid the full amount of their agreement for any project that 
disencumbered less than 50% of its funds. The remainder of the subcontractor contracts 
(those with >50% disencumbrance rate) were discounted by the disencumbrance rate 
actually experienced. Applying these factors to the reported subcontracts for training and 
administrative subcontracts, we determined that, under these assumptions, training 
subcontractors would have earned 67.7% of their original contracted amount and 
administrative subcontractors would have earned 51.3% of their original contracted 
amount. We applied these rates to the total estimated amount for subcontractors and 
consultants and obtained the following estimates for the high end of our range. 

Tr ainin g contracts: $41,217,000 X 67.7% = $27,904,000 [+ 13.7%] = $3 1,727,000 

Administrative contracts: $5,445,000 X 51.3% = 2,793,000 [+ 12.7%] = $3,148,000 
Maximum estimated subcontractor and consultant market 

for the two-year study period: $34,875,000 
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Our estimate puts the ETP-generated market for training and administrative subcontractors 
and consultants at roughly between $31 million and $35 million for the two-year study 
period. This estimate pertains to the entire market for subcontractors and consultants 
engendered by the full range of ETP contracts for the two years (the 394 contract database), 
and not just the subcontractors associated with ETP contracts with single employers (the 
260 contract database). 
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